Stereoscopic visualization camera and integrated robotics platform

ABSTRACT

A robotic imaging apparatus is disclosed. A robotic imaging apparatus includes a robotic arm, a stereoscopic camera, and a sensor positioned between the robotic arm and the stereoscopic camera. The sensor transmits output data that is indicative of translational and/or rotational force imparted on the stereoscopic camera by an operator. The robotic imaging apparatus also includes a processor that is configured to determine a movement sequence for the robotic arm based on a current position of the robotic arm and the output data from the sensor and to cause at least one of the joints of the robotic arm to rotate based on the determined movement sequence via one or more motor control signals provided to the at least one joint. The rotation of the at least one joint provides power-assisted movement of the robotic arm based on the detected translational and/or rotational forces imparted by the operator.

PRIORITY CLAIM

The present application claims priority to and the benefit as adivisional application of U.S. patent application Ser. No. 16/398,014,filed on Apr. 29, 2019, which claims priority to and the benefit of U.S.Provisional Patent Application No. 62/663,689, filed on Apr. 27, 2018,and which is a continuation-in-part of U.S. patent application Ser. No.15/814,127, filed Nov. 15, 2017, now U.S. Pat. No. 10,299,880, whichclaims priority to and the benefit of U.S. Provisional PatentApplication No. 62/489,289 filed on Apr. 24, 2017 and U.S. ProvisionalPatent Application No. 62/489,876 filed on Apr. 25, 2017, the entiretyof which are incorporated herein by reference.

BACKGROUND

Surgery is art. Accomplished artists create works of art that far exceedthe capabilities of a normal person. Artists use a brush to turncanisters of paint into vivid images that provoke strong and uniqueemotions from viewers. Artists take ordinary words written on paper andturn them into dramatic and awe-inspiring performances. Artists graspinstruments causing them to emit beautiful music. Similarly, surgeonstake seemingly ordinary scalpels, tweezers, and probes and producelife-altering biological miracles.

Like artists, surgeons have their own methods and preferences. Aspiringartists are taught the fundamentals of their craft. Beginners oftenfollow prescribed methods. As they gain experience, confidence, andknowledge, they develop their own unique artistry reflective ofthemselves and their personal environment. Similarly, medical studentsare taught the fundamentals of surgical procedures. They are rigorouslytested on these methods. As the students progress through residency andprofessional practice, they develop derivations of the fundamentals(still within medical standards) based on how they believe the surgeryshould best be completed. For instance, consider the same medicalprocedure performed by different renowned surgeons. The order of events,pacing, placement of staff, placement of tools, and use of imagingequipment varies between each of the surgeons based on theirpreferences. Even incision sizes and shapes can be unique to thesurgeon.

The artistic-like uniqueness and accomplishment of surgeons make themweary of surgical tools that change or alter their methods. The toolshould be an extension of the surgeon, operating simultaneously and/orin harmonious synchronization. Surgical tools that dictate the flow of aprocedure or change the rhythm of a surgeon are often discarded ormodified to conform.

In an example, consider microsurgery visualization where certainsurgical procedures involve patient structures that are too small for ahuman to visualize easily with the naked eye. For these microsurgeryprocedures, magnification is required to adequately view themicro-structures. Surgeons generally want visualization tools that arenatural extensions of their eyes. Indeed, early efforts at microsurgeryvisualization comprised attaching magnifying lens to head-mountedoptical eyepieces (called surgical loupes). The first pair was developedin 1876. Vastly improved versions of surgical loupes (some includingoptical zooms and integrated light sources) are still being used bysurgeons today. FIG. 1 shows a diagram of a pair of surgical loupes 100with a light source 102 and magnification lenses 104. The 150-yearstaying power of surgical loupes can be attributed to the fact that theyare literally an extension of a surgeon's eyes.

Despite their longevity, surgical loupes are not perfect. Loupes withmagnifying lenses and light sources, such as the loupes 100 of FIG. 1,have much greater weight. Placing even a minor amount of weight on thefront of a surgeon's face can increase discomfort and fatigue,especially during prolonged surgeries. The surgical loupes 100 alsoinclude a cable 106 that is connected to a remote power supply. Thecable effectively acts as a chain, thereby limiting the mobility of thesurgeon during their surgical performance.

Another microsurgery visualization tool is the surgical microscope, alsoreferred to as the operating microscope. Widespread commercialdevelopment of surgical microscopes began in the 1950s with theintention of replacing surgical loupes. Surgical microscopes includeoptical paths, lenses, and focusing elements that provide greatermagnification compared to surgical loupes. The large array of opticalelements (and resulting weight) meant that surgical microscopes had tobe detached from the surgeon. While this detachment gave the surgeonmore room to maneuver, the bulkiness of the surgical microscope causedit to consume considerable operating space above a patient, therebyreducing the size of the surgical stage.

FIG. 2 shows a diagram of a prior art surgical microscope 200. As onecan imagine, the size and presence of the surgical microscope in theoperating area made it prone to bumping. To provide stability andrigidity at the scope head 201, the microscope is connected torelatively large boom arms 202 and 204 or other similar supportstructure. The large boom arms 202 and 204 consume additional surgicalspace and reduce the maneuverability of the surgeon and staff. In total,the surgical microscope 200 shown in FIG. 2 could weigh as much as 350kilograms (“kg”).

To view a target surgical site using the surgical microscope 200, asurgeon looks directly though oculars 206. To reduce stress on asurgeon's back, the oculars 206 are generally positioned along asurgeon's natural line of sight using the arm 202 to adjust height.However, surgeons do not perform by only looking at a target surgicalsite. The oculars 206 have to be positioned such that the surgeon iswithin arm's length of a working distance to the patient. Such precisepositioning is critical to ensure the surgical microscope 200 becomes anextension rather than a hindrance to the surgeon, especially when beingused for extended periods of time.

Like any complex instrument, it takes surgeons tens to hundreds of hoursto feel comfortable using a surgical microscope. As shown in FIG. 2, thedesign of the surgical microscope 200 requires a substantially 90° angleoptical path from the surgeon to the target surgical site. For instance,a perfectly vertical optical path is required from the target surgicalsite to the scope head 201. This means that the scope head 201 has to bepositioned directly above the patient for every microsurgical procedure.In addition, the surgeon has to look almost horizontally (or some slightangle downward) into the oculars 206. A surgeon's natural inclination isto direct his vision to his hands at the surgical site. Some surgeonseven want to move their heads closer to the surgical site to have moreprecise control of their hand movements. Unfortunately, the surgicalmicroscopes 200 do not give surgeons this flexibility. Instead, surgicalmicroscopes 200 ruthlessly dictate that the surgeon is to place theireyes on the oculars 206 and hold their head at arm's length during theirsurgical performance, all while consuming valuable surgical space abovethe patient. A surgeon cannot even simply look down at a patient becausethe scope head 201 blocks the surgeon's view.

To make matters worse, some surgical microscopes 200 include a secondpair of oculars 208 for co-performers (e.g., assistant surgeons, nurses,or other clinical staff). The second pair of oculars 208 is usuallypositioned at a right angle from the first oculars 206. The closenessbetween the oculars 206 and 208 dictates that the assistant must stand(or sit) in close proximity to the surgeon, further restrictingmovement. This can be annoying to some surgeons who like to perform withsome space. Despite their magnification benefits surgical microscopes200 are not natural extensions of a surgeon. Instead, they areoverbearing directors in the surgical room.

SUMMARY

The present disclosure is directed to a stereoscopic robotic system thatincludes a stereoscopic visualization camera and robotic arm. Theexample stereoscopic robotic system is configured to acquirestereoscopic images of a target surgical site while enabling an operatorto position the stereoscopic visualization camera using the robotic arm.As disclosed herein, the robotic arm includes electro-mechanicallyoperated joints that provide structurally stability to enable thestereoscopic visualization camera to record high-resolution imageswithout jitter or other artifacts that can arise from unintended cameramovement. The robotic arm also provides structural flexibility thatpermits an operator to position the stereoscopic visualization camera atdifferent positions and/or orientations to obtain desired views of atarget surgical site. The example stereoscopic robotic systemaccordingly enables a surgeon to complete life-altering microsurgeriescomfortably in whatever position suits the surgeon.

The stereoscopic robotic system of the present disclosure can bepositioned about any number of orientations relative to the surgicalfield that best suits the needs of the surgeon or patient, rather thanthe physical and mechanical limitations of the visualization apparatus.The stereoscopic robotic system is configured to provide motorized jointmovement assistance that enables a surgeon or other operator toeffortlessly position the stereoscopic visualization camera. In someembodiments, the stereoscopic robotic system is configured to providemotorized assisted movement of a robotic arm based on forces detectedfrom an operator positioning the stereoscopic camera. The stereoscopicrobotic system may also enable an operator to select a visual lock on atarget surgical site while enabling the operator to change anorientation and/or position of the stereoscopic visualization camera.Additionally or alternatively, the stereoscopic robotic system isconfigured with one or more boundaries that prevent the stereoscopicvisualization camera and/or the robotic arm from contacting a patient,surgical staff, and/or surgical instruments. Altogether, thestereoscopic robotic system operates as an extension of a surgeon's eyeswhile giving the surgeon the freedom to conduct a microsurgery proceduregenerally without restrictions or impediments.

Aspects of the subject matter described herein may be useful alone or incombination with one or more other aspect described herein. Withoutlimiting the foregoing description, in a first aspect of the presentdisclosure, a robotic imaging apparatus includes a base sectionconfigured for connection to a secure structure or a cart and a roboticarm having a first end connected to the base section, a second endincluding a coupling interface, and a plurality of joints and linksconnecting the first end to the second end. Each joint includes a motorconfigured to rotate the joint around an axis and a joint sensorconfigured to transmit a position of the respective joint. The roboticimaging apparatus also includes a stereoscopic camera connected to therobotic arm at the coupling interface. The stereoscopic camera isconfigured to record left and right images of a target surgical site forproducing a stream of stereoscopic images of the target surgical site.The robotic imaging apparatus further includes a sensor positioned atthe coupling interface and configured to detect and transmit output datathat is indicative of translational and rotational force imparted on thestereoscopic camera by an operator. The robotic imaging apparatusadditionally includes a memory storing at least one algorithm defined byone or more instructions and/or data structures that specify a rotationdirection, speed, and duration for each of the joints of the robotic armbased at least on a current position of the robotic arm and detectedtranslational and rotational forces. Moreover, the robotic imagingapparatus includes at least one processor communicatively coupled to thesensor and the robotic arm. The at least one processor configured toreceive the output data from the sensor that is indicative of thetranslational and rotational forces and determine, using the at leastone algorithm in the memory, a movement sequence for the robotic armbased on a current position of the robotic arm and the output data fromthe sensor. The at least one processor is also configured to cause atleast one of the joints of the robotic arm to rotate based on thedetermined movement sequence via one or more motor control signalsprovided to the at least one joint. The rotation of the at least onejoint provides power-assisted movement of the robotic arm based on thedetected translational and rotational forces imparted on thestereoscopic camera by the operator.

In accordance with a second aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the at least one processor is configured to determine thecurrent position of the robotic arm based on output data from the jointsensors of the plurality of joints.

In accordance with a third aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the sensor includes at least one of a six-degrees-of-freedomhaptic force-sensing device or a torque sensor.

In accordance with a fourth aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the stereoscopic camera includes at least one control armhaving a release button to enable the power-assisted movement, and theat least one processor is configured to receive an input messageindicative that the release button was selected, and determine themovement sequence using the output data from the sensor after receivingthe input message related to the release button.

In accordance with a fifth aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the apparatus further includes a coupling plate with a firstend configured to connect to the coupling interface of the robotic armand a second end including a second coupling interface configured toconnect to the stereoscopic camera. The coupling plate includes at leastone joint including a joint sensor configured to transmit a position ofthe respective joint and a motor that is controllable by the at leastone processor according to the movement sequence.

In accordance with a sixth aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the sensor is located at the coupling interface or the secondcoupling interface.

In accordance with a seventh aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the coupling plate includes a second joint that enables thestereoscopic camera to be manually rotated by an operator between ahorizontal orientation and a vertical orientation. The second jointincludes a joint sensor configured to transmit a position of the secondjoint.

In accordance with an eighth aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the stereoscopic camera includes a housing including a bottomside that is configured to connect to the robotic arm at the couplinginterface.

In accordance with a ninth aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, a robotic imaging apparatus includes a robotic arm comprisinga first end for connection to a secure structure, a second end includinga coupling interface, and a plurality of joints and links connecting thefirst end to the second end, each joint including a motor configured torotate the joint around an axis and a joint sensor configured totransmit a position of the respective joint. The robotic imagingapparatus also includes an imaging device connected to the robotic armat the coupling interface, the imaging device configured to recordimages of a target surgical site, and a sensor positioned at thecoupling interface and configured to detect and transmit force and/ortorque output data that is indicative of force and/or torque imparted onthe imaging device by an operator. The robotic imaging apparatus furtherincludes at least one processor communicatively coupled to the sensorand the robotic arm. The at least one processor is configured to receivethe force and/or torque output data from the sensor, convert the forceand/or torque output data into translational and rotational vectors,determine, using kinematics, a movement sequence for the robotic armbased on a current position of the robotic arm and the translational androtational vectors, the movement sequence specifying a rotationdirection, a speed, and a duration of movement for at least some of thejoints of the robotic arm, and cause at least one of the joints of therobotic arm to rotate based on the determined movement sequence via oneor more motor control signals provided to the at least one joint.

In accordance with a tenth aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the processor is configured to determine a least one scalefactor based on at least one of the current position of the robotic armor a future position of the robotic arm based on the movement sequence,and apply the scale factor to at least one joint speed of the movementsequence.

In accordance with an eleventh aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the at least one scale factor is configured based on adistance of the robotic arm or the imaging device from a virtualboundary. The at least one scale factor decreases to a value of ‘0’ asthe virtual boundary is approached.

In accordance with a twelfth aspect of the present disclosure, which maybe used in combination with any other aspect listed herein unless statedotherwise, the virtual boundary corresponds to at least one of apatient, a medical instrument, or operating room staff.

In accordance with a thirteenth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the processor is configured to cause a display deviceto display an icon indicative that the at least one scale factor hasbeen applied to the movement sequence.

In accordance with a fourteenth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the processor is configured to determine a least onescale factor based on joint angles between joints of the robotic arm orjoint limits, and apply the scale factor to at least one joint speed ofthe movement sequence.

In accordance with a fifteenth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the processor is configured to provide gravitycompensation for the force and/or torque output data, and provideforce-application compensation for the force and/or torque output datato compensate for an offset between a location of the sensor and alocation of the imaging device upon which the force and/or torque isimparted by the operator.

In accordance with a sixteenth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the processor is configured to determine or identifyjoint singularities for the plurality of joints of the robotic arm forcontrol of hysteresis and backlash, and determine the movement sequencebased on the kinematics while avoiding robotic arm movement through thejoint singularities.

In accordance with a seventeenth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the robotic imaging apparatus further includes acoupling plate with a first end configured to connect to the couplinginterface of the robotic arm and a second end including a secondcoupling interface configured to connect to the stereoscopic camera. Thecoupling plate includes at least one joint including a joint sensorconfigured to transmit a position of the respective joint and a motorthat is controllable by the at least one processor according to themovement sequence. The sensor is located at the coupling interface orthe second coupling interface.

In accordance with an eighteenth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the robotic arm includes at least four joints and thecoupling plate includes at least two joints.

In accordance with a nineteenth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the processor is configured to cause at least one ofthe joints of the robotic arm to rotate by transmitting one or morecommand signals to the motor of the respective joint indicative of therotation direction, the speed, and the duration of movement as specifiedby the movement sequence.

In accordance with a twentieth aspect of the present disclosure, whichmay be used in combination with any other aspect listed herein unlessstated otherwise, the processor is configured to compare images recordedby the imaging device as the robotic arm is being moved during themovement sequence to confirm the robotic arm is being moved as specifiedduring the movement sequence.

In accordance with a twenty-first aspect of the present disclosure,which may be used in combination with any other aspect listed hereinunless stated otherwise, the kinematics includes at least one of inversekinematics or Jacobean kinematics.

In accordance with a twenty-second aspect of the present disclosure, anyof the structure and functionality illustrated and described inconnection with FIGS. 3 to 65 may be used in combination with any of thestructure and functionality illustrated and described in connection withany of the other of FIGS. 3 to 65 and with any one or more of thepreceding aspects.

In light of the aspects above and the disclosure herein, it isaccordingly an advantage of the present disclosure to provide astereoscopic robotic system that provides seamless coordination betweena stereoscopic camera and a robotic arm.

It is another advantage of the present disclosure to provide astereoscopic robotic system that uses a robotic arm to increase a focalrange, working distance, and/or magnification of a stereoscopic roboticsystem.

It is a further another advantage of the present disclosure to provide astereoscopic robotic system that provides powered assisted movement ofthe robotic arm based on forces/torques imparted on a stereoscopiccamera by an operator.

The advantages discussed herein may be found in one, or some, andperhaps not all of the embodiments disclosed herein. Additional featuresand advantages are described herein, and will be apparent from thefollowing Detailed Description and the figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows a diagram of a pair of prior art surgical loupes.

FIG. 2 shows a diagram of a prior art surgical microscope.

FIGS. 3 and 4 show diagrams of perspective views of a stereoscopicvisualization camera, according to an example embodiment of the presentdisclosure.

FIGS. 5 and 6 show diagrams of a microsurgical environment including thestereoscopic visualization camera of FIGS. 3 and 4, according to exampleembodiments of the present disclosure.

FIGS. 7 and 8 show diagrams illustrative of optical elements within theexample stereoscopic visualization camera of FIGS. 3 to 6, according toan example embodiment of the present disclosure.

FIG. 9 shows a diagram of a deflecting element of the examplestereoscopic visualization camera of FIGS. 7 and 8, according to anexample embodiment of the present disclosure.

FIG. 10 shows a diagram of an example of a right optical image sensorand a left optical image sensor of the example stereoscopicvisualization camera of FIGS. 7 and 8, according to an exampleembodiment of the present disclosure.

FIGS. 11 and 12 show diagrams of example carriers for optical elementsof the example stereoscopic visualization camera of FIGS. 7 and 8,according to example embodiments of the present disclosure.

FIG. 13 shows a diagram of an example flexure of the examplestereoscopic visualization camera of FIGS. 7 and 8, according to anexample embodiment of the present disclosure.

FIG. 14 shows a diagram of modules of the example stereoscopicvisualization camera for acquiring and processing image data, accordingto an example embodiment of the present disclosure.

FIG. 15 shows a diagram of internal components of the modules of FIG.14, according to an example embodiment of the present disclosure.

FIG. 16 shows a diagram of an information processor module of FIGS. 14and 15, according to an example embodiment of the present disclosure.

FIG. 17 shows an example of a display monitor, according to an exampleembodiment of the present disclosure.

FIGS. 18 to 21 show diagrams illustrative of spurious parallax betweenright and left optical paths.

FIG. 22 shows a diagram illustrative of an out-of-focus condition inrelation to a position of two parallel lenses for respective right andleft optical paths.

FIGS. 23 and 24 show diagrams illustrative of how spurious parallaxcauses digital graphics and/or images to lose accuracy when fused to astereoscopic image.

FIGS. 25 and 26 illustrate a flow diagram showing an example procedureto reduce or eliminate spurious parallax, according to an exampleembodiment of the present disclosure.

FIG. 27 shows a diagram illustrative of how a zoom repeat point isadjusted with respect to a pixel grid of an optical image sensor,according to an example embodiment of the present disclosure.

FIGS. 28 to 32 show diagrams illustrative of a template matching programto locate a zoom repeat point, according to an example embodiment of thepresent disclosure.

FIG. 33 shows a side-view of the microsurgical environment of FIG. 5,according to an example embodiment of the present disclosure.

FIG. 34 shows an embodiment of the example robotic arm of FIG. 5,according to an example embodiment of the present disclosure.

FIG. 35 shows a diagram of the robotic arm of FIGS. 33 and 34 connectedto a cart, according to an example embodiment of the present disclosure.

FIG. 36 shows a diagram where the robotic arm of FIGS. 33 and 34 ismounted to a ceiling plate, according to an example embodiment of thepresent disclosure.

FIG. 37 shows an embodiment of a coupling plate for the robotic arm,according to an example embodiment of the present disclosure.

FIGS. 38 to 40 show diagrams of the coupling plate in differentrotational positions, according to example embodiments of the presentdisclosure.

FIG. 41 illustrates an embodiment of the stereoscopic robotic platformof FIGS. 3 to 40, according to an example embodiment of the presentdisclosure.

FIG. 42 illustrates an example procedure or routine for calibrating thestereoscopic visualization camera of FIGS. 3 to 33, according to anexample embodiment of the present disclosure.

FIG. 43 shows an embodiment of the example stereoscopic visualizationcamera of FIGS. 3 to 33 and 42 moving an object plane in discrete steps,according to an example embodiment of the present disclosure.

FIG. 44 illustrates a graph illustrative of a routine executable by aprocessor for determining a center-of-projection of the stereoscopicvisualization camera of FIGS. 3 to 33 and 42, according to an exampleembodiment of the present disclosure.

FIG. 45 shows a plan view of an optical schematic that is illustrativeof how an interpupillary distance of the stereoscopic visualizationcamera of FIGS. 3 to 33 is measured and calibrated, according to anexample embodiment of the present disclosure.

FIG. 46 shows a plan view of an optical schematic that is illustrativeof how an optical axis of the stereoscopic visualization camera of FIGS.3 to 33 may be measured and calibrated, according to an exampleembodiment of the present disclosure.

FIG. 47 illustrates a diagram of a calibrated stereoscopic visualizationcamera in which optical parameters are fully characterized, according toan example embodiment of the present disclosure.

FIG. 48 illustrates an example procedure or routine for calibrating therobotic arm of FIGS. 5 and 33 to 41, according to an example embodimentof the present disclosure.

FIG. 49 shows a diagram that is illustrative of how the stereoscopicvisualization camera and/or the robotic arm are calibrated to robotspace, according to an example embodiment of the present disclosure.

FIG. 50 shows a diagram illustrative of horizontal and vertical boundaryplanes for restricting movement of the stereoscopic visualization cameraand/or the robotic arm, according to an example embodiment of thepresent disclosure.

FIG. 51 illustrates an example of how rotational joint speed of therobotic arm and/or the coupling plate is scaled based on distance to aboundary, according to an example embodiment of the present disclosure.

FIG. 52 shows a diagram of an example procedure for fusing an image froman alternate modality visualization with stereoscopic image(s),according to an example embodiment of the present disclosure.

FIGS. 53 to 61 show diagrams illustrative of live cross-sectional fusedvisualizations generated by the combination of the stereoscopicvisualization camera and/or the robotic arm of FIGS. 3 to 52, accordingto example embodiments of the present disclosure.

FIG. 62 shows a diagram that is illustrative of a procedure forproviding assisted drive of the stereoscopic visualization camera ofFIGS. 3 to 52, according to an example embodiment of the presentdisclosure.

FIG. 63 shows a diagram of an example procedure for moving the examplevisualization camera of FIGS. 3 to 52 using an input device, accordingto an example embodiment of the present disclosure.

FIG. 64 shows a diagram that is illustrative of an algorithm, routine,or procedure for providing a lock-to-target for the stereoscopicvisualization camera, according to an example embodiment of the presentdisclosure.

FIG. 65 shows a diagram that is illustrative of a virtual sphere for thelock-to-target feature of FIG. 64, according to an example embodiment ofthe present disclosure.

DETAILED DESCRIPTION

The present disclosure relates in general to a stereoscopicvisualization camera and platform. The stereoscopic visualization cameramay be referred to as a digital stereoscopic microscope (“DSM”). Theexample camera and platform are configured to integrate microscopeoptical elements and video sensors into a self-contained head unit thatis significantly smaller, lighter, and more maneuverable than prior artmicroscopes (such as the surgical loupes 100 of FIG. 1 and the surgicalmicroscope 200 of FIG. 2). The example camera is configured to transmita stereoscopic video signal to one or more television monitors,projectors, holographic devices, smartglasses, virtual reality devices,or other visual display devices within a surgical environment.

The monitors or other visual display devices may be positioned withinthe surgical environment to be easily within a surgeon's line of sightwhile performing surgery on a patient. This flexibility enables thesurgeon to place display monitors based on personal preferences orhabits. In addition, the flexibility and slim profile of thestereoscopic visualization camera disclosed herein reduces area consumedover a patient. Altogether, the stereoscopic visualization camera andmonitors (e.g., the stereoscopic visualization platform) enables asurgeon and surgical team to perform complex microsurgical surgicalprocedures on a patient without being dictated or restricted in movementcompared to the surgical microscope 200 discussed above. The examplestereoscopic visualization platform accordingly operates as an extensionof the surgeon's eyes, enabling the surgeon to perform masterpiecemicrosurgeries without dealing with the stress, restrictions, andlimitations induced by previous known visualization systems.

The disclosure herein generally refers to microsurgery. The examplestereoscopic visualization camera may be used in virtually anymicrosurgical procedure including, for example, cranial surgery, brainsurgery, neurosurgery, spinal surgery, ophthalmologic surgery, cornealtransplants, orthopedic surgery, ear, nose and throat surgery, dentalsurgery, plastics and reconstructive surgery, or general surgery.

The disclosure also refers herein to target site, scene, orfield-of-view. As used herein, target site or field-of-view includes anobject (or portion of an object) that is being recorded or otherwiseimaged by the example stereoscopic visualization camera. Generally thetarget site, scene, or field-of-view is a working distance away from amain objective assembly of the example stereoscopic visualization cameraand is aligned with the example stereoscopic visualization camera. Thetarget site may include a patient's biological tissue, bone, muscle,skin or combinations thereof. In these instances, the target site may bethree dimensional by having a depth component corresponding to aprogression of a patient's anatomy. The target site may also include oneor more templates used for calibration or verification of the examplestereoscopic visualization camera. The templates may be two-dimensional,such as a graphic design on paper (or plastic sheet) or threedimensional, such as to approximate a patient's anatomy in a certainregion.

Reference is also made throughout to an x-direction, a y-direction, az-direction, and a tilt-direction. The z-direction is along an axis fromthe example stereoscopic visualization camera to the target site andgenerally refers to depth. The x-direction and y-direction are in aplane incident to the z-direction and comprise a plane of the targetsite. The x-direction is along an axis that is 90° from an axis of they-direction. Movement along the x-direction and/or the y-direction referto in-plane movement and may refer to movement of the examplestereoscopic visualization camera, movement of optical elements withinthe example stereoscopic visualization camera, and/or movement of thetarget site.

The tilt-direction corresponds to movement along Euler angles (e.g., ayaw axis, a pitch axis, and a roll axis) with respect to thex-direction, the y-direction, and/or the z-direction. For example, aperfectly aligned lens has substantially a 0° tilt with respect to thex-direction, the y-direction, and/or the z-direction. In other words, aface of the lens is 90° or perpendicular to light along the z-direction.In addition, edges of the lens (if the lens has a rectangular shape) areparallel along the x-direction and the y-direction. Lens and/or opticalimage sensors can be titled through yaw movement, pitch movement, and/orroll movement. For example, a lens and/or optical image sensor may betitled along a pitch axis, with respect to the z-direction, to faceupwards or downwards. Light along the z-direction contacts a face of alens (that is pitched upwards or downwards) at non-perpendicular angle.Tilting of a lens and/or optical image sensor along a yaw axis, pitchaxis, or roll axis enables, for example, a focal point or ZRP to beadjusted.

I. Example Stereoscopic Visualization Camera

FIGS. 3 and 4 show diagrams of perspective views of a stereoscopicvisualization camera 300, according to an example embodiment of thepresent disclosure. The example camera 300 includes a housing 302configured to enclose optical elements, lens motors (e.g., actuators),and signal processing circuitry. The camera 300 has a width (along anx-axis) between 15 to 28 centimeters (cm), preferably around 22 cm. Inaddition, the camera 300 has a length (along a y-axis) between 15 to 32cm, preferably around 25 cm. Further, the camera 300 has a height (alonga z-axis) between 10 to 20 cm, preferably around 15 cm. The weight ofthe camera 300 is between 3 to 7 kg, preferably around 3.5 kg.

The camera 300 also includes control arms 304 a and 304 b (e.g.,operating handles), which are configured to control magnification level,focus, and other microscope features. The control arms 304 a and 304 bmay include respective controls 305 a and 305 b for activating orselecting certain features. For example, the control arms 304 a and 304b may include controls 305 a and 305 b for selecting a fluorescencemode, adjusting an amount/type of light projected onto a target site,and controlling a display output signal (e.g., selection between 1080por 4K and/or stereoscopic). In addition, the controls 305 a and/or 305 bmay be used to initiate and/or perform a calibration procedure and/ormove a robotic arm connected to the stereoscopic visualization camera300. In some instances, the controls 305 a and 305 b may include thesame buttons and/or features. In other instances the controls 305 a and305 b may include different features. Further, the control arms 304 aand 304 b may also be configured as grips to enable an operator toposition the stereoscopic visualization camera 300.

Each control arm 304 is connected to the housing 302 via a rotatablepost 306, as shown in FIG. 3. This connection enables the control arms304 to be rotated with respect to the housing 302. This rotationprovides flexibility to a surgeon to arrange the control arms 304 asdesired, further enhancing the adaptability of the stereoscopicvisualization camera 300 to be in synchronization with a surgicalperformance.

While the example camera 300 shown in FIGS. 3 and 4 includes two controlarms 304 a and 304 b, it should be appreciated that the camera 300 mayonly include one control arm or zero control arms. In instances wherethe stereoscopic visualization camera 300 does not include a controlarm, controls may be integrated with the housing 302 and/or provided viaa remote control.

FIG. 4 shows a bottom-up perspective view of a rear-side of thestereoscopic visualization camera 300, according to an exampleembodiment of the present disclosure. The stereoscopic visualizationcamera 300 includes a mounting bracket 402 configured to connect to asupport. As described in more detail in FIGS. 5 and 6, the support mayinclude an arm with one or more joints to provide significantmaneuverability. The arm may be connected to a movable cart or securedto a wall or ceiling.

The stereoscopic visualization camera 300 also includes a power port 404configured to receive a power adapter. Power may be received from an ACoutlet and/or a battery on a cart. In some instances, the stereoscopicvisualization camera 300 may include an internal battery to facilitateoperation without cords. In these instances, the power port 404 may beused to charge the battery. In alternative embodiments, the power port404 may be integrated with the mounting bracket 402 such that thestereoscopic visualization camera 300 receives power via wires (or otherconductive routing materials) within the support.

FIG. 4 also shows that the stereoscopic visualization camera 300 mayinclude a data port 406. The example data port 406 may include any typeof port including, for example, an Ethernet interface, a high-definitionmultimedia interface (“HDMI”) interface, a universal serial bus (“USB”)interface, a Serial Digital Interface (“SDI”), a digital opticalinterface, an RS-232 serial communication interface etc. The data port406 is configured to provide a communicative connection between thestereoscopic visualization camera 300 and cords routed to one or morecomputing devices, servers, recording devices, and/or display devices.The communicative connection may transmit stereoscopic video signals ortwo-dimensional video signals for further processing, storage, and/ordisplay. The data port 406 may also enable control signals to be sent tothe stereoscopic visualization camera 300. For instance, an operator ata connected computer (e.g., a laptop computer, desktop computer, and/ortablet computer) may transmit control signals to the stereoscopicvisualization camera 300 to direct operation, perform calibration, orchange an output display setting.

In some embodiments, the data port 406 may be replaced (and/orsupplemented) with a wireless interface. For example, the stereoscopicvisualization camera 300 may transmit stereoscopic display signals viaWi-Fi to one or more display devices. A use of a wireless interface,combined with an internal battery, enables the stereoscopicvisualization camera 300 to be wire-free, thereby further improvingmaneuverability within a surgical environment.

The stereoscopic visualization camera 300 shown in FIG. 4 also includesa front working distance main objective lens 408 of a main objectiveassembly. The example lens 408 is the start of the optical path withinthe stereoscopic visualization camera 300. Light from a light sourceinternal to the stereoscopic visualization camera 300 is transmittedthrough the lens 408 to a target site. Additionally, light reflectedfrom the target site is received in the lens 408 and passed todownstream optical elements.

II. Exemplary Maneuverability of the Stereoscopic Visualization Camera

FIGS. 5 and 6 show diagrams of the stereoscopic visualization camera 300used within a microsurgical environment 500, according to exampleembodiments of the present disclosure. As illustrated, the smallfootprint and maneuverability of the stereoscopic visualization camera300 (especially when used in conjunction with a multiple-degree offreedom arm) enables flexible positioning with respect to a patient 502.A portion of the patient 502 in view of the stereoscopic visualizationcamera 300 includes a target site 503. A surgeon 504 can position thestereoscopic visualization camera 300 in virtually any orientation whileleaving more than sufficient surgical space above the patient 502 (lyingin the supine position). The stereoscopic visualization camera 300accordingly is minimally intrusive (or not intrusive) to enable thesurgeon 504 to perform a life-altering microsurgical procedure withoutdistraction or hindrance.

In FIG. 5, the stereoscopic visualization camera 300 is connected to amechanical arm 506 via mounting bracket 402. The arm 506 may include oneor more rotational or extendable joints with electromechanical brakes tofacilitate easy repositioning of the stereoscopic visualization camera300. To move the stereoscopic visualization camera 300, the surgeon 504,or the assistant 508, actuates brake releases on one or more joints ofthe arm 506. After the stereoscopic visualization camera 300 is movedinto a desired position, the brakes may be engaged to lock the joints ofthe arm 506 in place.

A significant feature of the stereoscopic visualization camera 300 isthat it does not include oculars. This means that the stereoscopicvisualization camera 300 does not have to be aligned with the eyes ofthe surgeon 504. This freedom enables the stereoscopic visualizationcamera 300 to be positioned and orientated in desirable positions thatwere not practical or possible with prior known surgical microscopes. Inother words, the surgeon 504 can perform microsurgery with the mostoptimal view for conducting the procedure rather than being restrictedto merely adequate view dictated by oculars of a surgical microscope.

Returning to FIG. 5, the stereoscopic visualization camera 300, via themechanical arm 506, is connected to a cart 510 with display monitors 512and 514 (collectively a stereoscopic visualization platform orstereoscopic robotic platform 516). In the illustrated configuration,the stereoscopic visualization platform 516 is self-contained and may bemoved to any desired location in the microsurgical environment 500including between surgical rooms. The integrated platform 516 enablesthe stereoscopic visualization camera 300 to be moved and used on-demandwithout time needed to configure the system by connecting the displaymonitors 512 and 514.

The display monitors 512 and 514 may include any type of displayincluding a high-definition television, an ultra-high definitiontelevision, smart-eyewear, projectors, one or more computer screens,laptop computers, tablet computers, and/or smartphones. The displaymonitors 512 and 514 may be connected to mechanical arms to enableflexible positioning similar to the stereoscopic visualization camera300. In some instances, the display monitors 512 and 514 may include atouchscreen to enable an operator to send commands to the stereoscopicvisualization camera 300 and/or adjust a setting of a display.

In some embodiments, the cart 516 may include a computer 520. In theseembodiments, the computer 520 may control a robotic mechanical armconnected to the stereoscopic visualization camera 300. Additionally oralternatively, the computer 520 may process video (or stereoscopicvideo) signals (e.g., an image or frame stream) from the stereoscopicvisualization camera 300 for display on the display monitors 512 and514. For example, the computer 520 may combine or interleave left andright video signals from the stereoscopic visualization camera 300 tocreate a stereoscopic signal for displaying a stereoscopic image of atarget site. The computer 520 may also be used to store video and/orstereoscopic video signals into a video file (stored to a memory) so thesurgical performance can be documented and played back. Further, thecomputer 520 may also send control signals to the stereoscopicvisualization camera 300 to select settings and/or perform calibration.

In some embodiments, the microsurgical environment 500 of FIG. 5includes an ophthalmic surgery procedure. In this embodiment, themechanical arm 506 may be programmed to perform an orbiting sweep of apatient's eye. Such a sweep enables the surgeon to examine a peripheralretina during vitreo-retinal procedures. In contrast, with conventionaloptical microscopes, the only way a surgeon can view the peripheralretina is to push the side of the eye into the field of view using atechnique known as scleral depression.

FIG. 6 shows a diagram of the microsurgical environment 500 with thepatient 502 in a sitting position for a posterior-approach skull baseneurosurgery. In the illustrated embodiment, the stereoscopicvisualization camera 300 is placed into a horizontal position to facethe back of the head of the patient 502. The mechanical arm 506 includesjoints that enable the stereoscopic visualization camera 300 to bepositioned as shown. In addition, the cart 510 includes the monitor 512,which may be aligned with the surgeon's natural view direction.

The absence of oculars enables the stereoscopic visualization camera 300to be positioned horizontally and lower than the eye-level view of thesurgeon 504. Further, the relatively low weight and flexibility enablesthe stereoscopic visualization camera 300 to be positioned in waysunimaginable for other known surgical microscopes. The stereoscopicvisualization camera 300 thereby provides a microsurgical view for anydesired position and/or orientation of the patient 502 and/or thesurgeon 504.

While FIGS. 5 and 6 show two example embodiments for positioning thestereoscopic visualization camera 300, it should be appreciated that thestereoscopic visualization camera 300 may be positioned in any number ofpositions depending on the number of degrees of freedom of themechanical arm 506. It is entirely possible in some embodiments toposition the stereoscopic visualization camera 300 to face upwards(e.g., upside down).

III. Comparison of the Example Stereoscopic Visualization Platform toKnown Surgical Microscopes

In comparing the stereoscopic visualization camera 300 of FIGS. 3 to 6to the surgical microscope 200 of FIG. 2, the differences are readilyapparent. The inclusion of oculars 206 with the surgical microscoperequires that the surgeon constantly orient his/her eyes to eyepieces,which are in a fixed location relative to the scope head 201 andpatient. Further, the bulkiness and weight of the surgical microscoperestricts it to being positioned only in a generally verticalorientation with respect to a patient. In contrast, the examplestereoscopic visualization camera 300 does not include oculars and maybe positioned in any orientation or position with respect to a patient,thereby freeing the surgeon to move during surgery.

To enable other clinician staff to view a microsurgical target site, thesurgical microscope 200 requires the addition of second oculars 208.Generally, most known surgical microscopes 200 do not allow adding thirdoculars. In contrast, the example stereoscopic visualization camera 300may be communicatively coupled to an unlimited number of displaymonitors. While FIGS. 5 and 6 above showed display monitors 512 and 514connected to cart 510, a surgical room may be surrounded in displaymonitors that all show the microsurgical view recorded by thestereoscopic visualization camera 300. Thus, instead of limiting a viewto one or two people (or requiring sharing an ocular), an entiresurgical team can view a magnified view of a target surgical site.Moreover, people in other rooms, such as training and observation rooms,can be presented with the same magnified view displayed to the surgeon.

Compared to the stereoscopic visualization camera 300, the two-ocularsurgical microscope 200 is more prone to being bumped or inadvertentlymoved. Since surgeons place their heads on oculars 206 and 208 duringsurgery to look through eyepieces, the scope head 201 receives constantforce and periodic bumps. Adding the second oculars 208 doubles theforce from a second angle. Altogether, the constant force and periodicbumping by the surgeons may cause the scope head 201 to move, therebyrequiring the scope head 201 to be repositioned. This repositioningdelays the surgical procedure and annoys the surgeon.

The example stereoscopic visualization camera 300 does not includeoculars and is not intended to receive contact from a surgeon once it islocked into place. This corresponds to a significantly lower chance ofthe stereoscopic visualization camera 300 being accidentally moved orbumped during the surgeon's performance.

To facilitate the second oculars 208, the surgical microscope 200 has tobe outfitted with a beamsplitter 210, which may include glass lenses andmirrors housed in precision metallic tubes. The use of a beamsplitter210 reduces light received at the first oculars because some of thelight is reflected to the second oculars 208. Further, addition of thesecond oculars 208 and the beamsplitter 210 increases the weight andbulkiness of the scope head 201.

In contrast to the surgical microscope 200, the stereoscopicvisualization camera 300 only contains optical paths for sensors,thereby reducing weight and bulkiness. In addition, the optical sensorsreceive the full incident light since beamsplitters are not needed toredirect a portion of the light. This means the image received byoptical sensors of the example stereoscopic visualization camera 300 isas bright and clear as possible.

Some models of surgical microscopes may enable a video camera to beattached. For instance, the surgical microscope 200 of FIG. 2 includes amonoscopic video camera 212 connected to an optical path viabeamsplitter 214. The video camera 212 may be monoscopic orstereoscopic, such as the Leica® TrueVision® 3D Visualization SystemOphthalmology camera. The video camera 212 records an image receivedfrom the beamsplitter 214 for display on a display monitor. The additionof the video camera 212 and beamsplitter 214 further add to the weightof the scope head 201. In addition, the beamsplitter 214 consumesadditional light destined for the oculars 206 and/or 208.

Each beamsplitter 210 and 214 divides the incident light fractionallyinto three paths, removing light from the surgeon's view. The surgeon'seye has limited low-light sensitivity such that light from the operativesite presented to him/her must be sufficient to allow the surgeon toperform the procedure. However, a surgeon cannot always increase theintensity of light applied to a target site on a patient, especially inophthalmological procedures. A patient's eye has limited high-lightsensitivity before it develops light toxicity. Hence, there is alimitation to the number and fraction of beamsplitters and to the amountof light which can be split off from the first oculars 206 to enable theuse of ancillary devices 208 and 212.

The example stereoscopic visualization camera 300 of FIGS. 3 to 6 doesnot include beamsplitters such that optical imaging sensors receive thefull amount of light from a main objective assembly. This enables theuse of sensors with low-light sensitivity or even optical sensors withsensitivity outside the wavelengths of visible light to be used sincepost-processing can make the images sufficiently bright and visible (andadjustable) for display on the monitors.

Further, since the optical elements that define the optical paths areself-contained within the stereoscopic visualization camera 300, theoptical elements may be controlled through the camera. This controlallows placement and adjustment of the optical elements to be optimizedfor a three-dimensional stereoscopic display rather than for microscopeoculars. This configuration of the camera permits control to be providedelectronically from camera controls or from a remote computer. Inaddition, the control may be provided automatically through one or moreprograms onboard the camera 300 configured to adjust optical elementsfor retaining focus while zooming or to adjust for optical defectsand/or spurious parallax. In contrast, optical elements of the surgicalmicroscope 200 are external to the video camera 212 and controlled onlyvia operator input, which is generally optimized for viewing a targetsite through the oculars 206.

In a final comparison, the surgical microscope 200 includes an X-Ypanning device 220 for moving a field-of-view or target scene. The X-Ypanning device 220 is typically a large, heavy, and expensiveelectromechanical module since it must rigidly support and move thesurgical scope head 201. In addition, moving the scope head 201 changesthe positioning of the surgeon to the new location of the oculars 206.

In contrast, the example stereoscopic visualization camera 300 includesa memory including instructions, which when executed, cause a processorto select pixel data of optical sensors to enable X-Y panning across awide pixel grid. In addition, the example stereoscopic visualizationcamera 300 may include a small motor or actuator that controls a mainobjective optical element to change a working distance to a target sitewithout moving the camera 300.

IV. Example Optical Elements of the Stereoscopic Visualization Camera

FIGS. 7 and 8 show diagrams illustrative of optical elements within theexample stereoscopic visualization camera 300 of FIGS. 3 to 6, accordingto an example embodiment of the present disclosure. It may seemrelatively simple to acquire left and right views of a target site toconstruct a stereoscopic image. However, without careful design andcompensation, many stereoscopic images have alignment issues between theleft and right views. When viewed for a prolonged period of time,alignment issues can create confusion in an observer's brain as a resultof differences between the left and right views. This confusion can leadto headaches, fatigue, vertigo, and even nausea.

The example stereoscopic visualization camera 300 reduces (oreliminates) alignment issues by having a right optical path and leftoptical path with independent control and/or adjustment of some opticalelements while other left and right optical elements are fixed in acommon carrier. In an example embodiment, some left and right zoomlenses may be fixed to a common carrier to ensure left and rightmagnification is substantially the same. However, front or rear lensesmay be independently adjustable radially, rotationally, axially, and/ortilted to compensate for small differences in zoom magnification, visualdefects, and/or spurious parallax such as movement of a zoom repeatpoint. Compensation provided by adjustable lenses results in almostperfectly aligned optical paths throughout a complete zoom magnificationrange.

Additionally or alternatively, alignment issues may be reduced (oreliminated) using pixel readout and/or rendering techniques. Forexample, a right image (recorded by a right optical sensor) may beadjusted upwards or downwards with respect to a left image (recorded bya left optical sensor) to correct vertical misalignment between theimages. Similarly, a right image may be adjusted left or right withrespect to a left image to correct horizontal misalignment between theimages.

FIGS. 7 and 8 below show an example arrangement and positioning ofoptical elements that provide for almost artifact, spurious parallax,and distortion-free aligned optical paths. As discussed later, certainof the optical elements may be moved during calibration and/or use tofurther align the optical paths and remove any remaining distortions,spurious parallax, and/or defects. In the illustrated embodiment, theoptical elements are positioned in two parallel paths to generate a leftview and a right view. Alternative embodiments may include optical pathsthat are folded, deflected or otherwise not parallel.

The illustrated paths correspond to a human's visual system such thatthe left view and right view, as displayed on a stereoscopic display,appear to be separated by a distance that creates a convergence angle ofroughly 6 degrees, which is comparable to the convergence angle for anadult human's eyes viewing an object at approximately 4 feet away,thereby resulting in stereopsis. In some embodiments, image datagenerated from the left view and right view are combined together on thedisplay monitor(s) 512 and 514 to generate a stereoscopic image of atarget site or scene. Alternative embodiments comprise otherstereoscopic displays where the left view is presented to only the lefteye of a viewer and the corresponding right view is presented to onlythe right eye. In exemplary embodiments used to adjust and verify properalignment and calibration, both views are displayed overlaid to botheyes.

A stereoscopic view is superior to a monoscopic view because it mimicsthe human visual system much more closely. A stereoscopic view providesdepth perception, distance perception, and relative size perception toprovide a realistic view of a target surgical site to a surgeon. Forprocedures such as retinal surgery, stereoscopic views are vital becausesurgical movements and forces are so small that the surgeon cannot feelthem. Providing a stereoscopic view helps a surgeon's brain magnifytactile feel when the brain senses even minor movements while perceivingdepth.

FIG. 7 shows a side view of the example stereoscopic visualizationcamera 300 with the housing 302 being transparent to expose the opticalelements. FIG. 8 shows a diagram illustrative of an optical pathprovided by the optical elements shown in FIG. 7. As shown in FIG. 8,the optical path includes a right optical path and a left optical path.The optical paths in FIG. 8 are shown from a perspective of facing aforward direction and looking down at the stereoscopic visualizationcamera 300. From this view, the left optical path appear on the rightside of FIG. 8 while the right optical path is shown on the left side.

The optical elements shown in FIG. 7 are part of the left optical path.It should be appreciated that the right optical path in FIG. 7 isgenerally identical to the left optical path regarding relation locationand arrangement of optical elements. As mentioned above, theinterpupillary distance between a center of the optical paths is between58 to 70 mm, which may be scaled to 10 to 25 mm. Each of the opticalelements comprise lenses having certain diameters (e.g., between 2 mmand 29 mm). Accordingly, a distance between the optical elementsthemselves is between 1 to 23 mm, preferably around 10 mm.

The example stereoscopic visualization camera 300 is configured toacquire images of a target site 700 (also referred to as a scene orfield-of-view (“FOV”) or target surgical site). The target site 700includes an anatomical location on a patient. The target site 700 mayalso include laboratory biological samples, calibrationslides/templates, etc. Images from the target site 700 are received atthe stereoscopic visualization camera 300 via a main objective assembly702, which includes the front working distance lens 408 (shown in FIG.4) and a rear working distance lens 704.

A. Example Main Objective Assembly

The example main objective assembly 702 may include any type ofrefractive assembly or reflective assembly. FIG. 7 shows the objectiveassembly 702 as an achromatic refractive assembly with the front workingdistance lens 408 being stationary and the rear working distance lens704 being movable along the z-axis. The front working distance lens 408may comprise a plano convex (“PCX”) lens and/or a meniscus lens. Therear working distance lens 704 may comprise an achromatic lens. Inexamples where the main objective assembly 702 includes an achromaticrefractive assembly, the front working distance lens 408 may include ahemispherical lens and/or a meniscus lens. In addition, the rear workingdistance lens 704 may include an achromatic doublet lens, an achromaticdoublet group of lenses, and/or an achromatic triplet lens.

The magnification of the main objective assembly 702 is between 6× to20×. In some instances, the magnification of the main objective assembly702 may vary slightly based on a working distance. For example, the mainobjective assembly 702 may have a magnification of 8.9× for a 200 mmworking distance and a magnification of 8.75× for a 450 mm workingdistance.

The example rear working distance lens 704 is configured to be movablewith respect to the front working distance lens 408 to change a spacingtherebetween. The spacing between the lenses 408 and 704 determines theoverall front focal length of the main objective assembly 702, andaccordingly the location of a focal plane. In some embodiments, thefocal length is the distance between the lenses 408 and 704 plusone-half the thickness of the front working distance lens 408.

Together, the front working distance lens 408 and the rear workingdistance lens 704 are configured to provide an infinite conjugate imagefor providing an optimal focus for downstream optical image sensors. Inother words, an object located exactly at the focal plane of the targetsite 700 will have its image projected at a distance of infinity,thereby being infinity-coupled at a provided working distance.Generally, the object appears in focus for a certain distance along theoptical path from the focal plane. However, past the certain thresholddistance, the object begins to appear fuzzy or out of focus.

FIG. 7 shows working distance 706, which is the distance between anouter surface of the front working distance lens 408 and to the focalplane of the target site 700. The working distance 706 may correspond toan angular field-of-view, where a longer working distance results in awider field-of-view or larger viewable area. The working distance 706accordingly sets a plane of the target site or scene that is in focus.In the illustrated example, the working distance 706 is adjustable from200 to 450 mm by moving the rear working distance lens 704. In anexample, the field-of-view can be adjusted between 20 mm×14 mm to 200mm×140 mm using upstream zooming lenses when the working distance is 450mm.

The main objective assembly 702 shown in FIGS. 7 and 8 provides an imageof the target site 700 for both the left and right optical paths. Thismeans that the width of the lenses 408 and 704 should be at least aswide as the left and right optical paths. In alternative embodiments,the main objective assembly 702 may include separate left and rightfront working distance lenses 408 and separate left and right rearworking distance lens 704. The width of each pair of the separateworking distance lenses may be between ¼ to ½ of the width of the lenses408 and 704 shown in FIGS. 7 and 8. Further, each of the rear workingdistance lenses 704 may be independently adjustable.

In some embodiments, the main objective assembly 702 may be replaceable.For example, different main objective assemblies may be added to changea working distance range, a magnification, a numerical aperture, and/orrefraction/reflection type. In these embodiments, the stereoscopicvisualization camera 300 may change positioning of downstream opticalelements, properties of optical image sensors, and/or parameters ofimage processing based on which main objective assembly is installed. Anoperator may specify which main objective assembly is installed in thestereoscopic visualization camera 300 using one of the controls 305 ofFIG. 3 and/or a user input device.

B. Example Lighting Sources

To illuminate the target site 700, the example stereoscopicvisualization camera 300 includes one or more lighting sources. FIGS. 7and 8 show three lighting sources including a visible light source 708a, a near-infrared (“NIR”) light source 708 b, and a near-ultraviolet(“NUV”) light source 708 c. In other examples, the stereoscopicvisualization camera 300 may include additional or fewer (or no) lightsources. For instance, the NIR and NUV light sources may be omitted. Theexample light sources 708 are configured to generate light, which isprojected to the target scene 700. The generated light interacts andreflects off the target scene, with some of the light being reflected tothe main objective assembly 702. Other examples may include externallight sources or ambient light from the environment.

The example visible light source 708 a is configured to output light inthe human-visible part of the light spectrum in addition to some lightwith wavelengths outside the visible region. The NIR light source 708 bis configured to output light that is primarily at wavelengths slightlypast the red part of the visible spectrum, which is also referred to as“near-infrared.” The NUV light source 708 c is configured to outputlight that is primarily at wavelengths in the blue part of the visiblespectrum, which is referred to as “near-ultraviolet.” The light spectraoutput by the light sources 708 is controlled by respective controllers,described below. A brightness of light emitted by the light sources 708may be controlled by a switching rate and/or applied voltage waveform.

FIGS. 7 and 8 illustrate that the visible light source 708 a and the NIRlight source 708 b are provided directly through the main objectiveassembly 702 to the target site 700. As shown in FIG. 8, visible lightfrom the visible light source 708 a propagates along visible path 710 a.Additionally, NIR light from the NIR light source 708 b propagates alongNIR path 710 b. While the light sources 708 a and 708 b are shown asbeing behind the main objective assembly 702 (with respect to the targetsite 700), in other examples the light sources 708 a and 708 b may beprovided before the main objective assembly 702. In one embodiment, thelight sources 708 a and 708 b may be provided on an outside of thehousing 302 and face toward the target site 700. In yet otherembodiments, the light sources 708 may be provided separate from thestereoscopic visualization camera 300 using, for example, a Koeherillumination setup and/or a darkfield illumination setup.

In contrast to the light sources 708 a and 708 b, NUV light from the NUVlight source 708 c is reflected by a deflecting element 712 (e.g., abeamsplitter) to the main objective assembly 702 using anepi-illumination setup. The deflecting element 712 may be coated orotherwise configured to reflect only light beyond the NUV wavelengthrange, thereby filtering NUV light. NUV light from the NUV light source708 c propagates along NUV path 710 c.

In some embodiments, the NIR and NUV light sources 708 b and 708 c maybe used with excitation filters to further filter light that may not beblocked by filters (e.g., filter 740). The filters may be placed infront of the light sources 708 b and 708 c before the main objectiveassembly 702 and/or after the main objective assembly. The light fromthe NUV and NIR light sources 708 b and 708 c, after being filtered,comprises wavelengths that excite fluorescence in fluorescent sites 914(shown in FIG. 9) of an anatomical object. Further, the light from theNUV and NIR light sources 708 b and 708 c, after being filtered, maycomprise wavelengths that are not in the same range as those beingemitted by the fluorescent sites 914.

The projection of the light from light sources 708 through the mainobjective assembly provides the benefit of changing the lightedfield-of-view based on the working distance 706 and/or focal plane.Since the light passes through the main objective assembly 702, theangle at which light is projected changes based on the working distance706 and corresponds to the angular field-of-view. This configurationaccordingly ensures the field-of-view is properly illuminated by thelight sources 708, regardless of working distance or magnification.

C. Example Deflecting Element

The example deflecting element 712 illustrated in FIGS. 7 and 8 isconfigured to transmit a certain wavelength of light from the NUV lightsource 708 c to the target site 700 through the main objective assembly702. The deflecting element 712 is also configured to reflect lightreceived from the target site 700 to downstream optical elements,including a front lens set 714 for zooming and recording. In someembodiments, the deflecting element 712 may filter light received fromthe target site 700 through the main objective assembly 702 so thatlight of certain wavelengths reaches the front lens set 714.

The deflecting element 712 may include any type of mirror or lens toreflect light in a specified direction. In an example, the deflectingelement 712 includes a dichroic mirror or filter, which has differentreflection and transmission characteristics at different wavelengths.The stereoscopic visualization camera 300 of FIGS. 7 and 8 includes asingle deflecting element 712, which provides light for both the rightand left optical paths. In other examples, the camera 300 may includeseparate deflecting elements for each of the right and left opticalpaths. Further, a separate deflecting element may be provided for theNUV light source 708 c.

FIG. 9 shows a diagram of the deflecting element 712 of FIGS. 7 and 8,according to an example embodiment of the present disclosure. Forbrevity, the main objective assembly 702 is not shown. In this example,the deflecting element 712 includes two parallel faces 902 and 904 fortransmitting and reflecting light of certain wavelengths. The parallelfaces 902 and 904 are set at a 45° angle with respect to the left andright optical paths (represented as path 906). The 45° angle is selectedsince this angle causes reflected light to propagate at a 90° angle fromthe transmitted light, thereby providing optimal separation withoutcausing the separated light to be detected in the downstream front lensset 714. In other embodiments, the angle of the deflecting element 712could be between 10 degrees and 80 degrees without unintentionallypropagating light of unwanted wavelengths.

The example NUV light source 708 c is located behind the deflectingelement 712 (with respect to the target site 700). Light from the lightsource 708 c propagates along path 908 and contacts the deflectingelement 712. NUV light around the primary wavelength range of the NUVlight source 708 c is transmitted through the deflecting element 712along path 910 to the target site 700. Light from the NUV light source708 c that has a wavelength above (and below) the primary wavelengthrange of the NUV light source 708 c is reflected along path 912 to alight sink or unused region of the housing 302.

When the NUV light reaches the target site 700, it is absorbed by one ormore fluorescent sites 914 of an anatomical object. In some instances,the anatomical object may have been injected with a contrast agentconfigured to absorb NUV light and emit light with a different primarywavelength. In other instances, the anatomical object may naturallyabsorb NUV light and emit light with a different primary wavelength. Atleast some of the light reflected or emitted by the fluorescent site 914propagates along path 916 until it contacts the deflecting element 712.Most of the light reflects off the surface 904 along path 906 to thefront lens set 714. A portion of the light, including NUV light aroundthe primary wavelength range of the NUV light source 708 c istransmitted through the deflecting element 712 along path 918 to a lightsink or unused region of the housing 302. The deflecting element 712shown in FIG. 9 accordingly enables optical stimulation of a fluorescentagent at the target site 700 with one region of the spectrum whileblocking much of the stimulating light from travelling to the downstreamfront lens set 714.

It should be appreciated that the reflectivity and transmissivitycharacteristics of the deflecting element 712 can be changed to meetother light spectrum requirements. In some instances, the housing 302may include a slot that enables the deflecting element 712 and/or theNUV light source 708 c to be replaced based on the desired lightreflectivity and transmissivity characteristics. It should also beappreciated that a first path internal to the deflecting element 712between path 908 and path 910 and a second path internal to thedeflecting element 712 between path 916 and path 918 are each angled torepresent schematically the refraction of the light as it travelsbetween air and the interior of the deflecting element 712. The anglesshown are not meant to represent actual reflection angles.

D. Example Zoom Lenses

The example stereoscopic visualization camera 300 of FIGS. 7 and 8includes one or more zoom lens to change a focal length and angle ofview of the target site 700 to provide zoom magnification. In theillustrated example, the zoom lens includes the front lens set 714, azoom lens assembly 716, and a lens barrel set 718. It should beappreciated that in other embodiments, the front lens set 714 and/or thelens barrel set 718 may be omitted. Alternatively, the zoom lens mayinclude additional lens to provide further magnification and/or imageresolution.

The front lens set 714 includes a right front lens 720 for the rightoptical path and a left front lens 722 for the left optical path. Thelenses 720 and 722 may each include a positive converging lens to directlight from the deflecting element 712 to respective lenses in the zoomlens assembly 716. A lateral position of the lenses 720 and 722accordingly defines a beam from the main objective assembly 702 and thedeflecting element 712 that is propagated to the zoom lens assembly 716.

One or both of the lenses 720 and 722 may be adjustable radially tomatch optical axes of the left and right optical paths. In other words,one or both of the lenses 720 and 722 may be moved left-right and/orup-down in a plane incident to the optical path. In some embodiments,one or more of the lenses 720 and 722 may be rotated or tilted to reduceor eliminate image optical defects and/or spurious parallax. Movingeither or both of the lenses 720 and 722 during zooming may cause thezoom repeat point (“ZRP”) for each optical path to appear to remainstationary to a user. In addition to radial movement, one or both of thefront lenses 720 and 722 may be moved axially (along the respectiveoptical path) to match magnifications of the optical paths.

The example zoom lens assembly 716 forms an afocal zoom system forchanging the size of a field-of-view (e.g., a linear field-of-view) bychanging a size of the light beam propagated to the lens barrel set 718.The zoom lens assembly 716 includes a front zoom lens set 724 with aright front zoom lens 726 and a left front zoom lens 728. The zoom lensassembly 716 also includes a rear zoom lens set 730 with a right rearzoom lens 732 and a left rear zoom lens 734. The front zoom lenses 726and 728 may be positive converging lenses while the rear zoom lenses 732and 734 include negative diverging lenses.

The size of an image beam for each of the left and right optical pathsis determined based on a distance between the front zoom lenses 726 and728, the rear zoom lenses 732 and 734 and the lens barrel set 718.Generally, the size of the optical paths reduces as the rear zoom lenses732 and 734 move toward the lens barrel set 718 (along the respectiveoptical paths), thereby decreasing magnification. In addition, the frontzoom lenses 726 and 728 may also move toward (or away from) the lensbarrel set 718 (such as in a parabolic arc), as the rear zoom lenses 732and 734 move toward the lens barrel set 718, to maintain the location ofthe focal plane on the target site 700, thereby maintaining focus.

The front zoom lenses 726 and 728 may be included within a first carrier(e.g., the front zoom set 724) while the rear zoom lenses 732 and 724are included within a second carrier (e.g., the rear zoom set 730). Eachof the carriers 724 and 730 may be moved on tracks (or rails) along theoptical paths such that left and right magnification changesconcurrently. In this embodiment, any slight differences inmagnification between the left and right optical paths may be correctedby moving the right front lens 720 and/or the left front lens 722.Additionally or alternatively, a right lens barrel 736 and/or a leftlens barrel 738 of the lens barrel set 718 may be moved axially.

In alternative embodiments, the right front zoom lens 726 may be movedaxially separately from the left front zoom lens 728. In addition, theright rear zoom lens 732 may be moved axially separately from the leftrear zoom lens 734. Separate movement may enable small magnificationdifferences to be corrected by the zoom lens assembly 716, especiallywhen the front lens set 714 and the lens barrel set 718 are stationaryalong the optical paths. Further, in some embodiments, the right frontzoom lens 726 and/or the left front zoom lens 728 may be radially and/orrotationally adjustable (and/or tilted) to maintain an apparent locationof a ZRP in the optical path. Additionally or alternatively, the rightrear zoom lens 732 and/or the left rear zoom lens 734 may be radiallyand/or rotationally adjustable (and/or tilted) to maintain an apparentlocation of a ZRP in the optical path.

The example lens barrel set 718 includes the right lens barrel 736 andthe left lens barrel 738, which are part of the afocal zoom system inaddition with the zoom lens assembly 716. The lenses 736 and 738 mayinclude positive converging lenses configured to straighten or focus alight beam from the zoom lens assembly 716. In other words, the lenses736 and 738 focus the infinity-coupled output of the zoom lens assembly716.

In some examples, the lens barrel set 718 is fixed radially and axiallywithin the housing 302. In other examples, the lens barrel set 718 maybe movable axially along the optical path to provide increasedmagnification. Additionally or alternatively, each of the lenses 736 and738 may be radially and/or rotationally adjustable (and/or tilted) to,for example, correct for differences in optical properties (frommanufacturing or natural glass deviations) between the left and rightlenses of the front lens set 714, the front zoom lens set 724, and/orthe rear zoom lens set 730.

Altogether, the example front lens set 714, the zoom lens assembly 716,and the lens barrel set 718 are configured to achieve an optical zoombetween 5× to about 20×, preferably at a zoom level that hasdiffraction-limited resolution. In some embodiments, the front lens set714, the zoom lens assembly 716, and the lens barrel set 718 may providehigher zoom ranges (e.g., 25× to 100×) if image quality can becompromised. In these embodiments, the stereoscopic visualization camera300 may output a message to an operator indicative that a selectedoptical range is outside of an optical range and subject to a reductionin image quality.

In some embodiments, the lenses of the front lens set 714, the zoom lensassembly 716, the lens barrel set 718, and/or the main objectiveassembly 702 may each be constructed as a doublet from multiple opticalsub-elements using materials that balance each other's opticaldistortion parameters. The doublet construction reduces chromaticaberrations and optical aberrations. For example, the front workingdistance lens 408 and the rear working distance lens 702 may each beconstructed as a doublet. In another example, the front lenses 720 and722, the front zoom lenses 726 and 728, the rear zoom lenses 732 and734, and the lens barrels 736 and 738 may each comprise a doublet lens.

In yet further embodiments, the lenses of the front lens set 714, thezoom lens assembly 716, the lens barrel set 718, and/or the mainobjective assembly 702 may be tuned differently and/or have differentproperties to provide two parallel optical paths with differentcapabilities. For example, right lenses in zoom lens assembly 716 may beselected to provide 5× to 10× optical zoom for the right optical pathwhile left lenses in the zoom lens assembly 716 are selected to provide15× to 20× optical zoom for the left optical path. Such a configurationmay enable two different magnifications to be shown at the same timeand/or on the same screen, though in a monoscopic view.

E. Example Filter

The example stereoscopic visualization camera 300 of FIGS. 7 and 8includes one or more optical filters 740 (or filter assemblies) toselectively transmit desired wavelengths of light. FIG. 8 shows that asingle filter 740 may be applied to the right and left optical paths. Inother examples, each of the optical paths may have a separate filter.The inclusion of separate filters enables, for example, differentwavelengths of light to be filtered from the left and right opticalpaths at the same time, which enables, for example, fluorescent imagesto be displayed in conjunction with visible light images.

FIG. 7 shows that the filter 740 includes a wheel that is rotated aboutits axis of rotation. In the illustrated embodiment, the filter 740 canaccommodate three different optical filter pairs. However, in otherembodiments, the filter 740 may include additional or fewer filterpairs. Generally, light received at the filter 740 from the target site700 includes a broad spectrum of wavelengths. The lenses of the mainobjective assembly 702, the front lens set 714, the zoom lens assembly716, and the lens barrel set 718 are configured to pass a relativelywide bandwidth of light including wavelengths of interest to an operatorand undesirable wavelengths. In addition, downstream optical imagesensors are sensitive to certain wavelengths. The example filter 740accordingly passes and blocks certain portions of the light spectrum toachieve different desirable features.

As a wheel, the filter 740 comprises a mechanical device capable ofchanging positions at about four times per second. In other embodiments,the filter 740 may include a digital micro-mirror, which can change alight path's direction at video frame rates such as 60 times per second.In these other embodiments, each of the left and right optical pathswould include a micro-mirror. The left and right micro-mirror may havesynchronized or simultaneous switching.

In some embodiments, the filter 740 may be synchronized to the lightsources 708 to realize “time-interleaved” multispectral imaging. Forexample, the filter 740 may include an infrared cut filter,near-infrared bandpass filter, and near-ultraviolet cut filter. Thedifferent filter types are selected to work with different spectra ofthe light sources 708 and the reflectivity and transmissivitycharacteristics of the deflecting element 712 to pass certain desiredwavelengths of light at predetermined times.

In one mode, the filter 740 and the light sources 708 are configured toprovide a visible light mode. In this mode, the visible light source 708a transmits light from the visible region onto the target site 700, someof which is reflected to the main objective assembly 702. The reflectedlight may include some light beyond the visible spectrum, which mayaffect optical image sensors. The visible light is reflected by thedeflecting element 712 and passes through the front lens set 714, thezoom lens assembly 716, and the lens barrel set 718. In this example,the filter 740 is configured to apply the infrared-cut filter or thenear-ultraviolet cut filter to the optical paths to remove light outsidethe visible spectrum such that light only in the visible spectrum passesthrough to a final optical set 742 and an optical image sensor 744.

In another mode, filter 740 and the light sources 708 are configured toprovide fluorescence light of a narrow wavelength to the optical sensor744. In this mode, the NUV light source 708 c transmits light from thedeep-blue region of the spectrum to the target site 700. The deflectingelement 712 allows the desired light of the deep-blue region to passthrough while reflecting undesired light. The deep-blue light interactswith the target site 700 such that fluorescence light is emitted. Insome examples, δ-Aminolaevulinic acid (“5ala”) and/or Protoporphyrin IXis applied to the target site 700 to cause fluorescence light to beemitted when deep-blue light is received. The main objective assembly702 receives the fluorescence light in addition to reflected deep-bluelight and some visible light. The deep-blue light passes through thedeflecting element 712 out of the right and left optical paths. Thus,only the visible light and fluorescence light pass through the frontlens set 714, the zoom lens assembly 716, and the lens barrel set 718.In this example, the filter 740 is configured to apply thenear-ultraviolet cut filter to the optical paths to remove light outsidethe desired fluorescence spectrum including visible light and anyremaining NUV deep-blue light. Accordingly, only fluorescence light of anarrow wavelength reaches the optical image sensor 744, which enablesthe fluorescence light to be more easily detected and distinguishedbased on relative intensity.

In yet another mode, the filter 740 and the light sources 708 areconfigured to provide indocyanine green (“ICG”) fluorescence light tothe optical sensor 744. In this mode, the NIV light source 708 btransmits light in the far-red region (which is also considerednear-infrared) of the visible spectrum to the target site 700. Inaddition, the visible light source 708 a transmits visible light to thetarget scene 700. The visible light and far-red light are absorbed bymaterial with ICG at the target site, which then emits a highlystimulated fluorescence light in the further-red region. The mainobjective assembly 702 receives the fluorescence light in addition toreflected NIR light and visible light. The light is reflected by thedeflecting element 712 to the front lens set 714, the zoom lens assembly716, and the lens barrel set 718. In this example, the filter 740 isconfigured to apply the near-infrared bandpass filter to the opticalpaths to remove light outside the desired fluorescence spectrumincluding visible light and at least some of the NIR light. Accordingly,only fluorescence light in the further-red region reaches the opticalimage sensor 744, which enables the fluorescence light to be more easilydetected and distinguished based on relatively intensity.

TABLE 1 Light Transmitted to Light Source Filter Image Sensors VisibleInfrared Cut Filter, Visible Light Near-Ultraviolet Cut Filter NUVNear-Ultraviolet Cut Filter Blue Visible and NIR Light NIR andNear-Infrared Bandpass Filter Further-Red Fluorescence Visible

Table 1 above shows a summary of the different possible combinations oflights sources and filters for causing light of a certain desiredwavelength to reach the optical light sensor 744. It should beappreciated that other types of filters and/or light sources may be usedto further increase the different types of light received at the imagesensor 744. For instance, bandpass filters configured to pass light of anarrow wavelength may be used to correspond to certain biological stainsor contrasts applied to the target site 700. In some examples, thefilter 740 may include a cascade or more than one filter to enable lightfrom two different ranges to be filtered. For example, a first filter740 may apply an infrared cut filter and a near-ultraviolet cut filtersuch that only visible light of a desired wavelength range passes to theoptical sensor 744.

In other embodiments, separate filters 740 may be used for the left andright optical paths. For example, a right filter may include an infraredcut filter while a left filter includes a near-infrared pass filter.Such a configuration enables viewing of the target site 700 in visiblewavelengths simultaneously with IGC green fluorescence wavelengths. Inanother example, a right filter may include an infrared cut filter whilea left filter includes a near-ultraviolet cut filter. In thisconfiguration, the target site 700 may be shown in visible lightsimultaneously with SALA fluorescence light. In these other embodiments,the right and left image streams may still be combined into astereoscopic view that provides a fluorescence view of certainanatomical structures combined with a view of the target site 700 invisible light.

F. Example Final Optical Element Set

The example stereoscopic visualization camera 300 of FIGS. 7 and 8includes the final optical element set 742 to focus light received fromthe filter 740 onto the optical image sensor 744. The final opticalelement set 742 includes a right final optical element 745 and a leftfinal optical element 747, which may each comprise a positive converginglens. In addition to focusing light, the optical elements 745 and 747may be configured to correct minor aberrations in the right and leftoptical paths prior to the light reaching the optical image sensor 744.In some examples, the lenses 745 and 747 may be movable radially and/oraxially to correct magnification and/or focusing aberrations caused bythe front lens set 714, the zoom lens assembly 716, and the lens barrelset 718. In an example, the left final optical element 747 may be movedradially while the right final optical element 745 is fixed to removeZRP movement during magnification changes.

G. Example Image Sensors

The example stereoscopic visualization camera 300 of FIGS. 7 and 8includes the image sensor 744 to acquire and/or record incident lightthat is received from the final optical element set 742. The imagessensor 744 includes a right optical image sensor 746 to acquire and/orrecord light propagating along the right optical path and a left opticalimage sensor 748 to acquire and/or record light propagating along theleft optical path. Each of the left and right optical image sensors 746and 748 include, for example, complementary metal-oxide-semiconductor(“CMOS”) sensing elements, N-type metal-oxide-semiconductor (“NMOS”),and/or semiconductor charge-coupled device (“CCD”) sensing elements. Insome embodiments, the left and right optical sensors 746 and 748 areidentical and/or have the same properties. In other embodiments, theleft and right optical sensors 746 and 748 include different sensingelements and/or properties to provide varying capability. For example,the right optical image sensor 746 (using a first color filter array)may be configured to be more sensitive to blue fluorescence light whilethe left optical image sensor 748 (using a second color filter array) isconfigured to be more sensitive to visible light.

FIG. 10 shows an example of the right optical image sensor 746 and theleft optical image sensor 748 of the image sensor 744, according to anexample embodiment of the present disclosure. The right optical imagesensor 746 includes a first two-dimensional grid or matrix 1002 oflight-sensing elements (e.g., pixels). In addition, the left opticalimage sensor 748 includes a second two-dimensional pixel grid 1004 oflight-sensing elements. Each of the pixels includes a filter thatenables only light of a certain wavelength to pass, thereby contactingan underlying light detector. Filters for different colors are spreadacross the sensors 746 and 748 to provide light detection for allwavelengths across grids. The light detector may be sensitive to visiblelight, as well as additional ranges that are above and below the visiblespectrum.

The light-sensing elements of the grids 1002 and 1004 are configured torecord a range of wavelengths of light as a representation of the targetsite 700 that is in the field-of-view. Light incident on a light-sensingelement causes an electrical change to accumulate. The electrical chargeis read to determine an amount of light being received at the sensingelement. In addition, since the filter characteristics of the sensingelement are known to within manufacturing tolerances, the range ofwavelengths of the received light is known. The representation of thetarget site 700 is directed onto the light-sensing elements such thatthe grids 1002 and 1004 for the respective optical image sensors 746 and748 sample the target site 700 spatially. The resolution of the spatialsampling is a parameter that affects image quality and parity.

The number of pixels shown in the pixel grids 1002 and 1004 in FIG. 10is not representative of the number of actual pixels in the opticalimage sensors 746 and 748. Instead, the sensors typically have aresolution between 1280×720 pixels and 8500×4500 pixels, preferablyaround 2048×1560 pixels. However, not all pixels of the grids 1002 and1004 are selected for image transmission. Instead, a subset or pixel setof the grids 1002 and 1004 are selected for transmission. For example,in FIG. 10, pixel set 1006 is selected from the pixel grid 1002 fortransmission as a right image and pixel set 1008 is selected from pixelgrid 1004 for transmission as a left image. As illustrated, the pixelset 1006 does not need to be located in the same location as the pixelset 1008 in relation to respective pixel grids 1002 and 1004. Theseparate control of the pixel sets 1006 and 1008 enables left and rightimages to be aligned and/or corrected for image defects and/or spuriousparallax such as moving ZRPs.

Selection of a pixel set from a pixel grid enables a portion of thepixel grid to be selected to compensate for image defects/spuriousparallax and/or to more align the right and left optical images. Inother words, the pixel set may be moved or adjusted (in real-time) withrespect to the pixel grid to improve image quality by reducing oreliminating spurious parallax. Alternatively, either or both of the leftand right views of the stereoscopic image can be moved virtually in theimage processing pipeline (for example during rendering of the views fordisplay) to accomplish the same effect. Rotational misalignment of thesensors can also be corrected virtually. A pixel set may also be movedacross a pixel grid during use to provide an appearance of panning thefield-of-view. In an example, a pixel set or window of 1920×1080 pixelsmay be selected from a pixel grid having 2048×1560 pixels. The locationof the pixel window or set may be controlled by software/firmware and bemoved during setup and/or use. The resolution of the optical imagesensors 746 and 748 is accordingly specified based on a number of pixelsin the length and width directions of the pixel set or window.

1. Color Sensing with the Example Image Sensors

As mentioned above, the optical sensing elements 746 and 748 includepixels with different filters to detect certain colors of light. Forinstance, some pixels are covered with filters that pass predominantlyred light, some are covered with filters that pass predominantly greenlight, and some are covered with filters that pass predominantly bluelight. In some embodiments, a Bayer pattern is applied to the pixelgrids 1002 and 1004. However, it should be appreciated that in otherembodiments, a different color pattern may be used that is optimized forcertain wavelengths of light. For example, a green filter in eachsensing region may be replaced with a broadband filter or anear-infrared filter, thereby extending the sensing spectrum.

The Bayer pattern is implemented by grouping two rows by two columns ofpixels and covering one with a red filter, one with a blue filter, andtwo with a green filter, each in a checkerboard pattern. Thus theresolution of red and blue are each one quarter of the whole sensingregion of interest while green resolution is half that of the wholesensing region of interest.

Green may be assigned to half the sensing region to cause the opticalimage sensors 746 and 748 to operate as a luminance sensor and mimic thehuman visual system. In addition, red and blue mimic chrominance sensorsof the human visual system, but are not as critical as green sensing.Once an amount of red, green, and blue are determined for a certainregion, other colors in the visible spectrum are determined by averagingthe red, green, and blue values, as discussed in conjunction withde-Bayer program 1580 a of FIG. 16 discussed below.

In some embodiments, the optical image sensors 746 and 748 may usestacked components to sense color rather than filters. For example,sensing elements may include red, green and blue sensing componentsstacked vertically inside a pixel's area. In another example, prismssplit incident light into components using specially coatedbeamsplitters one or more times (typically at least two times resultingin three component colors, known as “3-chip”) with sensing elementsplaced in each of the split beams' paths. Other sensor types use adifferent pattern such as replacing one of the green filters with abroadband filter or a near-infrared filter, thereby extending thesensing possibilities of the digital surgical microscope.

2. Sensing Light Outside the Visible Range with the Example ImageSensors

The example sensing element filters of the optical image sensors 746 and748 are configured to also pass near-infrared light in a range that thesensing element can detect. This enables the optical image sensors 746and 748 to detect at least some light outside of the visible range. Suchsensitivity may decrease image quality in the visible part of thespectrum because it “washes out” the image, reducing contrast in manytypes of scenes and negatively affecting the color quality. As a result,the filter 740 may use the infrared cut filter to block near infraredwavelengths while passing the visible wavelengths to the optical imagesensors 746 and 748.

However, such near-infrared sensitivity may be desirable. For example, afluorescent agent, such ICG, can be introduced to the target site 700.ICG becomes excited or activated with visible or other wavelengths orlight and emits fluorescence light in the near infrared range. Asmentioned above, the NIR light source 708 b provides NIR light and thevisible light source 708 a provides visible light to excite agents withICG. Emitted light is further along the red spectrum, which may bepassed through the filter 740 using a near-infrared bandpass orhigh-pass filter. The light from the red spectrum then is detected bythe optical image sensors 746 and 748. By matching the spectralcharacteristics of the filter 740 to the expected behaviors of the lightsource 708 and the fluorescent agent, the agent and the biologicalstructures, such as blood that contain the agent, can be differentiatedat the target site 700 from other structures that do not contain theagent.

Note that in this example, the NIR light source 708 b has a differentprimary wavelength from the near-infrared filter in the filter 740.Specifically, the NIR light source 708 b has a primary wavelength around780 nanometers (“nm”) (around which the majority of the light's outputspectrum exists). In contrast, the near-infrared filter of the filter740 transmits light at wavelengths in a range of approximately 810 nm to910 nm. The light from the NIR light source 708 b and light passedthrough the filter 740 are both “near-infrared” wavelengths. However,the light wavelengths are separated so that the example stereoscopicvisualization camera 300 can stimulate with the light source 708 anddetect with the optical image sensor 744 while filtering the stimulationlight. This configuration accordingly enables the use of fluorescentagents.

In another embodiment, agents can be excited in the blue, violet, andnear-ultraviolet region and fluoresce light in the red region. Anexample of such an agent includes porphyrin accumulation in malignantgliomas caused by the introduction of SALA. In this example, it isnecessary to filter out the blue light while passing the remainder ofthe spectrum. A near-ultraviolet cut filter is used for this situation.As in the case with “near-infrared” discussed above, the NUV lightsource 708 c has a different primary wavelength from thenear-ultraviolet cut filter in the filter 740.

H. Example Lens Carrier

Section IV(D) above mentions that at least some of the lenses of thefront lens set 714, the zoom lens assembly 716, and/or the lens barrelset 718 may move in one or more carriers along rails. For example, thefront zoom lens set 724 may comprise a carrier that moves front zoomlens 726 and 728 together axially.

FIGS. 11 and 12 show diagrams of example carriers, according to exampleembodiments of the present disclosure. In FIG. 11, carrier 724 includesthe right front zoom lens 726 and the left front zoom lens 728 within asupport structure 1102. The carrier 724 includes a rail holder 1104configured to moveably connect to rail 1106. A force ‘F’ is applied toan actuation section 1108 to cause the carrier 724 to move along therail 1106. The force ‘F’ may be applied by a leadscrew or other linearactuation device. As illustrated in FIG. 11, the force ‘F’ is applied atan offset of the carrier 724. Friction between the rail 1106 and thecarrier 724 generates a moment M_(y) that causes the support structure1102 to move slightly around the Y-axis shown in FIG. 11. This slightmovement may cause the right front zoom lens 726 and the left front zoomlens 728 to shift slightly in opposite directions causing spuriousparallax, which is an error in a parallax between views of astereoscopic image.

FIG. 12 shows another example of the carrier 724. In this example, force‘F’ is applied symmetrically at center structure 1202, which isconnected to the rail holder 1104 and the support structure 1102. Theforce ‘F’ generates a moment M_(x) that causes the carrier 724 to rotateor move slightly around the X-axis shown in FIG. 12. The rotationalmovement causes the right front zoom lens 726 and the left front zoomlens 728 to shift in the same direction by the same degree of movement,thereby reducing (or eliminating) the onset of spurious parallax.

While FIGS. 11 and 12 show lenses 726 and 728 within one carrier, inother embodiments the lenses 726 and 728 may each be within a carrier.In these examples, each lens would be on a separate track or rail.Separate leadscrews may be provided for each of the lenses to provideindependent axial movement along the respective optical path.

I. Example Flexure

Section IV(D) above mentions that at least some of the lenses of thefront lens set 714, the zoom lens assembly 716, and/or the lens barrelset 718 may be moved radially, rotated, and/or tilted. Additionally oralternatively, the optical image sensors 746 and 748 may be movedaxially and/or tilted with respect to their respective incident opticalpath. The axial and/or tilt movement may be provided by one or moreflexures. In some examples, the flexures may be cascaded such that afirst flexure provides motion in a first direction and separate flexureprovides independent motion in a second direction. In another example, afirst flexure provides tilt along a pitch axis and separate flexureprovides tilt along a yaw axis.

FIG. 13 shows a diagram of an example dual flexure 1300, according to anexample embodiment of the present disclosure. The flexure 1300illustrated in FIG. 13 is for the optical image sensor 744 and isconfigured to independently move the right optical image sensor 746 andthe left optical image sensor 748 along their respective optical axisfor purposes of final focusing. The flexure 1300 includes a support beam1301 for connection to the housing 302 of the example stereoscopicvisualization camera 300 and to provide a rigid base for actuation. Theflexure 1300 also includes a beam 1302 for each channel (e.g., sensor746 and 748) that is rigid in all directions except for the direction ofmotion 1310. The beam 1302 is connected to flexing hinges 1303 thatenable the beam 1302 to move in a direction of motion 1310, aparallelogram translation in this example.

An actuator device 1304 flexes the beam 1302 in the desired directionfor a desired distance. The actuator device 1304 includes a push-screw1306 and a pull screw 1308, for each channel, which apply oppositeforces to the beam 1302 causing the flexing hinges 1303 to move. Thebeam 1302 may be moved inward, for example, by turning the push-screw1306 to push on the beam 1302. The flexure 1300 illustrated in FIG. 13is configured to independently move the right optical image sensor 746and the left optical image sensor 748 axially along their optical axis.

After the beam 1302 is flexed into a desired position, a lockingmechanism is engaged to prevent further movement, thereby creating arigid column. The locking mechanism includes the push-screw 1306 and itsrespective concentric pull screw 1308, that when tightened, create largeopposing forces that result in the rigid column of the beam 1302.

While the optical image sensors 746 and 748 are shown as being connectedto the same flexure 1300, in other examples, the sensors may beconnected to separate flexures. For example, returning to FIG. 8, theright optical image sensor 746 is connected to flexure 750 and the leftoptical image sensor 748 is connected to flexure 752. The use of theseparate flexures 750 and 752 enables the optical image sensors 746 and748 to be separately adjusted to, for example, align the left and rightoptical views and/or reduce or eliminate spurious parallax.

In addition, while FIG. 13 shows image sensors 746 and 748 connected tothe flexure 1300, in other examples, the lenses of the front lens set714, the zoom lens assembly 716, the lens barrel set 718, and/or thefinal optical element set 742 may be connected to alternative oradditional flexures instead. In some instances, each of the right andleft lenses of the front lens set 714, the zoom lens assembly 716, thelens barrel set 718, and/or the final optical element set 742 may beconnected to a separate flexure 1300 to provide independent radial,rotational, and/or tilt adjustment.

The flexure 1300 may provide motion resolution of less than a micron. Asa result of the very fine motion adjustment, images from the right andleft optical paths may have an alignment accuracy of several or even onepixel for a 4K display monitor. Such accuracy is viewed on each display512, 514 by overlaying the left and right views and observing both viewswith both eyes, rather than stereoscopically.

In some embodiments, the flexure 1300 can include the flexure disclosedin U.S. Pat. No. 5,359,474, titled “SYSTEM FOR THE SUB-MICRONPOSITIONING OF A READ/WRITE TRANSDUCER,” the entirety of which isincorporated herein by reference. In yet other embodiments, the lensesof the front lens set 714, the zoom lens assembly 716, the lens barrelset 718, and/or the final optical element set 742 may be stationary in aradial direction. Instead, a deflecting element (e.g., a mirror) with anadjustable deflection direction in an optical path may be used to steerthe right and/or left optical paths to adjust alignment and/or spuriousparallax. Additionally or alternatively, a tilt/shift lens may beprovided in the optical path. For instance, a tilt of an optical axismay be controlled with an adjustable wedge lens. In further embodiments,lenses of the front lens set 714, the zoom lens assembly 716, the lensbarrel set 718, and/or the final optical element set 742 may includedynamic lenses with parameters that can be changed electronically. Forexample, the lenses may include Varioptic liquid lenses produced byInvenios France SAS.

V. Example Processors of the Stereoscopic Visualization Camera

The example stereoscopic visualization camera 300 is configured torecord image data from the right and left optical paths and output theimage data to the monitor(s) 512 and/or 514 for display as astereoscopic image. FIG. 14 shows a diagram of modules of the examplestereoscopic visualization camera 300 for acquiring and processing imagedata, according to an example embodiment of the present disclosure. Itshould be appreciated that the modules are illustrative of operations,methods, algorithms, routines, and/or steps performed by certainhardware, controllers, processors, drivers, and/or interfaces. In otherembodiments, the modules may be combined, further partitioned, and/orremoved. Further, one or more of the modules (or portions of a module)may be provided external to the stereoscopic visualization camera 300such as in a remote server, computer, and/or distributed computingenvironment.

In the illustrated embodiment of FIG. 14, the components 408, 702 to750, and 1300 in FIGS. 7 to 13 are collectively referred to as opticalelements 1402. The optical elements 1402 (specifically the optical imagesensors 746 and 748) are communicatively coupled to an image capturemodule 1404 and a motor and lighting module 1406. The image capturemodule 1404 is communicatively coupled to an information processormodule 1408, which may be communicatively coupled to an externallylocated user input device 1410 and one or more display monitors 512and/or 514.

The example image capture module 1404 is configured to receive imagedata from the optical image sensors 746 and 748. In addition, the imagecapture module 1404 may define the pixel sets 1006 and 1008 within therespective pixel grids 1002 and 1004. The image capture module 1404 mayalso specify image recording properties, such as frame rate and exposuretime.

The example motor and lighting module 1406 is configured to control oneor more motors (or actuators) to change a radial, axial, and/or tiltposition of one or more of the optical elements 1402. For instance, amotor or actuator may turn a drive screw to move the carrier 724 alongthe track 1106, as shown in FIGS. 11 and 12. A motor or actuator mayalso turn the push-screw 1306 and/or the pull screw 1308 of the flexure1300 of FIG. 13 to adjust a radial, axial, or tilt position of a lensand/or optical image sensor. The motor and lighting module 1406 may alsoinclude drivers for controlling the light sources 708.

The example information processor module 1408 is configured to processimage data for display. For instance, the information processor module1408 may provide color correction to image data, filter defects from theimage data, and/or render image data for stereoscopic display. Theinformation processor module 1408 may also perform one or morecalibration routines to calibrate the stereoscopic visualization camera300 by providing instructions to the image capture module 1404 and/orthe motor and lighting module 1406 to perform specified adjustments tothe optical elements. The information processor module 1408 may furtherdetermine and provide in real-time instructions to the image capturemodule 1404 and/or the motor and lighting module 1406 to improve imagealignment and/or reduce spurious parallax.

The example user input device 1410 may include a computer to provideinstructions for changing operation of the stereoscopic visualizationcamera 300. The user input device 1410 may also include controls forselecting parameters and/or features of the stereoscopic visualizationcamera 300. In an embodiment, the user input device 1410 includes thecontrol arms 304 of FIG. 3. The user input device 1410 may be hardwiredto the information processor module 1408. Additionally or alternatively,the user input device 1410 is wirelessly or optically communicativelycoupled to the information processor module 1408.

The example display monitors 512 and 514 include, for example,televisions and/or computer monitors configured to provide athree-dimensional viewing experience. For example, the display monitorsmay include the LG® 55LW5600 television. Alternatively, the displaymonitors 512 and 514 may include a laptop screen, tablet screen, asmartphone screen, smart-eyewear, a projector, a holographic display,etc.

The sections that follow describe the image capture module 1404, themotor and lighting module 1406, and the information processor module1408 in more detail.

A. Example Image Capture Module

FIG. 15 shows a diagram of the image capture module 1404, according toan example embodiment of the present disclosure. The example imagecapture module 1404 includes an image sensor controller 1502, whichincludes a processor 1504, a memory 1506, and a communications interface1508. The processor 1504, the memory 1506, and the communicationsinterface 1508 may be communicatively coupled together via an imagesensor controller bus 1512.

The processor 1504 is programmable with one or more programs 1510 thatare persistently stored within the memory 1506. The programs 1510include machine readable instructions, which when executed, cause theprocessor 1504 to perform one or more steps, routines, algorithms, etc.In some embodiments, the programs 1510 may be transmitted to the memory1506 from the information processor module 1408 and/or from the userinput device 1410. In other examples, the programs 1510 may betransmitted to the processor 1504 directly from the informationprocessor module 1408 and/or from the user input device 1410.

The example image sensor controller 1502 is communicatively coupled tothe right optical image sensor 746 and the left optical image sensor 748of the optical elements 1402. The image sensor controller 1502 isconfigured to provide power to the optical image sensors 746 and 748 inaddition to sending timing control data and/or programming data. Inaddition, the image sensor controller 1502 is configured to receiveimage and/or diagnostic data from the optical image sensors 746 and 748.

Each of the optical image sensors 746 and 748 contains programmableregisters to control certain parameters and/or characteristics. One ormore of the registers may specify a location of the pixel sets 1006 and1008 within the respective pixel grids 1002 and 1004 of FIG. 10. Theregisters may store a value of a starting location with respect to anorigin point or edge point of the pixel grids 1002 and 1004. Theregisters may also specify a width and height of the pixel sets 1006 and1008 to define a rectangular region of interest. The image sensorcontroller 1502 is configured to read pixel data for pixels that arewithin the specified pixel sets 1006 and 1008. In some embodiments, theregisters of the optical image sensors 746 and 748 may facilitate thedesignation of pixel sets of other shapes, such as circles, ovals,triangles, etc. Additionally or alternatively, the registers of theoptical image sensors 746 and 748 may enable multiple pixel sets to bespecified simultaneously for each of the pixel grids 1002 and 1004.

A light-sensing portion of the pixels of the pixel grids 1002 and 1004is controlled by embedded circuitry, which specifies different modes oflight-sensing. The modes include a reset mode, an integration mode, anda readout mode. During the reset mode, a charge storage component of apixel is reset to a known voltage level. During the integration mode,the pixel is switched to an “on” state. Light that reaches a sensingarea or element of the pixel causes a charge to accumulate in a chargestorage component (e.g., a capacitor). The amount of stored electricalcharge corresponds to the amount of light incident on the sensingelement during the integration mode. During the readout mode, the amountof electrical charge is converted into a digital value and read out ofthe optical image sensors 746 and 748 via the embedded circuitry andtransmitted to the image sensor controller 1502. To read every pixel,the charge storage component of each pixel in a given region isconnected sequentially by switched internal circuitry to a readoutcircuit, which performs the conversion of the electrical charge from ananalog value to digital data. In some embodiments, the pixel analog datais converted to 12-bit digital data. However, it should be appreciatedthat the resolution may be less or greater based on allowances fornoise, settling time, frame rate, and data transmission speed. Thedigital pixel data of each pixel may be stored to a register.

The example processor 1504 of the image sensor controller 1502 of FIG.15 is configured to receive pixel data (e.g., digital data indicative ofan electrical charge stored in the pixel corresponding to an amount ofincident light on an element of the pixel) from each of the pixelswithin the pixel sets 1006 and 1008. The processor 1504 forms a rightimage from the pixel data received from the right optical image sensor746. In addition, the processor 1504 forms a left image from the pixeldata received from the left optical image sensor 748. Alternatively, theprocessor 1504 forms only a portion (for example, one row or severalrows) of each the left and right images before transmitting the datadownstream. In some embodiments, the processor 1504 uses a registerlocation to determine a location of each pixel within an image.

After the right and left images are created, the processor 1504synchronizes the right and left images. The processor 1504 thentransmits both of the right and left images to the communicationsinterface 1508, which processes the images into a format fortransmission to the information processor module 1408 via acommunications channel 1514. In some embodiments, the communicationschannel 1514 conforms to the USB 2.0 or 3.0 standard and may comprise acopper or fiber optical cable. The communications channel 1514 mayenable up to approximately 60 pairs (or more) of left and right images(having a stereoscopic resolution of 1920×1080 and a data conversionresolution of 12-bits) per second to be transmitted per second. The useof a copper USB cable enables power to be provided from the informationprocessor module 1408 to the image capture module 1404.

The sections below further describe features provided by the processor1504 of the image sensor controller 1502 executing certain programs 1510to acquire and/or process image data from the optical image sensors 746and 748.

1. Exposure Example

The example processor 1504 may control or program an amount of time theoptical image sensors 746 and 748 are in the integration mode, discussedabove. The integration mode occurs for a time period referred to as anexposure time. The processor 1504 may set the exposure time by writing avalue to an exposure register of the optical image sensors 746 and 748.Additionally or alternatively, the processor 1504 may transmitinstructions to the optical image sensors 746 and 748 signaling thestart and end of the exposure time. The exposure time may beprogrammable between a few milliseconds (“ms”) to a few seconds.Preferably the exposure time is approximately the inverse of the framerate.

In some embodiments, the processor 1504 may apply a rolling shuttermethod to the optical image sensors 746 and 748 to read pixel data.Under this method, the exposure time for a given row of pixels of thepixel sets 1006 and 1008 begins just after the pixels in that row havebeen read out and then reset. A short time later, the next row (which istypically physically most proximate to the row just set) is read, andaccordingly reset with its exposure time restarted. The sequentialreading of each pixel row continues until the last or bottom row of thepixel sets 1006 and 1008 have been read and reset. The processor 1504then returns to the top row of the pixel sets 1006 and 1008 to readpixel data for the next image.

In another embodiment, the processor 1504 applies a global shuttermethod. Under this method, the processor 1504 implements readout andreset in a manner similar to the rolling shutter method. However, inthis method integration occurs simultaneously for all pixels in thepixel sets 1006 and 1008. The global shutter method has the advantage ofreducing defects in an image compared to the rolling shutter methodsince all of the pixels are exposed at the same time. In comparison, inthe rolling shutter method, there is a small time delay between exposingthe lines of the pixel set. Small defects can develop during the timesbetween line exposures, especially between top lines and bottom lineswhere small changes at the target site 700 between reads can occur.

2. Dynamic Range Example

The example processor 1504 may execute one or more programs 1510 todetect light that is outside of a dynamic range of the optical imagesensors 746 and 748. Generally, extremely bright light completely fillsa charge storage region of a pixel, thereby resulting in lost imageinformation regarding the exact brightness level. Similarly, extremelylow light or lack of light fails to impart a meaningful charge in apixel, which also results in lost image information. Images created fromthis pixel data accordingly do not accurately reflect the lightintensity at target site 700.

To detect light that is outside the dynamic range, the processor 1504may execute one of several high dynamic range (“HDR”) programs 1510including, for example, a multiple-exposure program, a multi-slope pixelintegration program, and a multi-sensor image fusion program. In anexample, the multiple-exposure program may utilize HDR featuresintegrated or embedded with the optical image sensors 746 and 748. Underthis method, the pixel sets 1006 and 1008 are placed into theintegration mode for a normal expose time. The lines of the pixel sets1006 and 1008 are read and stored in a memory at the optical imagesensors 746 and 748 and/or the memory 1506 of the image sensorcontroller 1502. After the read is performed by the processor 1504, eachline in the pixel sets 1006 and 1008 is turned on again for a secondexposure time that is less than the normal exposure time. The processor1504 reads each of the lines of pixels after the second exposure timeand combines this pixel data with the pixel data from the normalexposure time for the same lines. The processor 1504 may applytone-mapping to choose between (or combine) the pixel data from thenormal-length and short-length exposure times and map the resultingpixel data to a range that is compatible with downstream processing anddisplay. Using the multiple-exposure program, the processor 1504 is ableto expand the dynamic range of the optical image sensors 746 and 748 andcompress the resulting range of pixel data for display.

The processor 1510 may operate a similar program for relatively darklight. However, instead of the second exposure time being less than thenormal time, the second exposure time is greater than the normal time,thereby providing the pixels more time to accumulate a charge. Theprocessor 1510 may use tone-mapping to adjust the read pixel data tocompensate for the longer exposure time.

3. Frame Rate Example

The example processor 1510 may control or specify a frame rate for theoptical image sensors 746 and 748. In some embodiments, the opticalimage sensors 746 and 748 include on-board timing circuitry andprogrammable control registers to specify the number of times per secondeach of the pixels within the pixel sets 1006 and 1008 are to be cycledthrough the imaging modes discussed above. A frame or image is formedeach time the pixel set progresses through the three modes. A frame rateis the number of times per second the pixels in the pixel sets 1006 and1008 are integrated, read, and reset.

The processor 1510 may be synchronized with the optical image sensors746 and 748 such that reads are conducted at the appropriate time. Inother examples, the processor 1510 is asynchronous with the opticalimage sensors 746 and 748. In these other examples, the optical imagesensors 746 and 748 may store pixel data after a local read to atemporary memory or queue. The pixel data may then be read periodicallyby the processor 1510 for right and left image synchronization.

The processing of frames or images in a time-sequential manner (e.g.,creation of an image stream) provides an illusion of motion conveyed asa video. The example processor 1510 is configured to program a framerate that provides the appearance of a smooth video to an observer. Aframe rate that is too low makes any motion appear choppy or uneven.Movie quality above a maximum threshold frame rate is not discernible toan observer. The example processor 1510 is configured to generateapproximately 20 to 70 frames per second, preferably between 50 and 60frames per second for typical surgical visualization.

4. Sensor Synchronization Example

The example processor 1504 of FIG. 15 is configured to control thesynchronization of the optical image sensors 746 and 748. The processor1504 may, for instance, provide power simultaneously to the opticalimage sensors 746 and 748. The processor 1504 may then provide a clocksignal to both of the optical image sensors 746 and 748. The clocksignal enables the optical image sensors 746 and 748 to operateindependently in a free-run mode but in a synchronized and/orsimultaneous manner. Accordingly, the optical image sensors 746 and 748record pixel data at nearly the same time. The example processor 1504receives the pixel data from the optical image sensors 746 and 748,constructs at least a fraction of the images and/or frames andsynchronizes the images and/or frames (or fraction thereof) to accountfor any slight timing mismatches. Typically, the lag between the opticalimage sensors 746 and 748 is less than 200 microseconds. In otherembodiments, the processor 1504 may use a synchronization pin tosimultaneously activate the optical image sensors 746 and 748 after, forexample, each reset mode.

B. Example Motor and Lighting Module

The example stereoscopic visualization camera 300 of FIG. 15 includesthe motor and lighting module 1406 to control one or more motors oractuators for moving lenses of the optical elements 1402 and/orcontrolling lighting output from the light sources 708. The examplemotor and lighting module 1406 includes a motor and lighting controller1520 that contains a processor 1522, a memory 1524, and a communicationsinterface 1526 that are communicatively coupled together viacommunication bus 1528. The memory 1524 stores one or more programs 1530that are executable on the processor 1522 to perform control,adjustment, and/or calibration of the lenses of the optical elements1402 and/or the light sources 708. In some embodiments, the programs1530 may be transmitted to the memory 1524 from the informationprocessor module 1408 and/or the user input device 1410.

The communications interface 1526 is communicatively coupled to thecommunications interface 1508 of the image capture module 1404 and acommunications interface 1532 of the information processor module 1408.The communications interface 1526 is configured to receive commandmessages, timing signals, status messages, etc. from the image capturemodule 1404 and the information processor module 1408. For example, theprocessor 1504 of the image capture module 1404 may send timing signalsto the processor 1522 to synchronize timing between lighting control andexposure time of the optical image sensors 746 and 748. In anotherexample, the information processing module 1408 may send commandmessages instructing certain light sources 708 to be activated and/orcertain lenses of the optical elements 1402 to be moved. The commandsmay be in response to input received from an operator via, for example,the user input device 1410. Additionally or alternatively, the commandsmay be in response to a calibration routine and/or real-time adjustmentto reduce or eliminate image misalignment and/or defects such asspurious parallax.

The example motor and lighting module 1406 includes drivers that providepower to control motors for adjusting an axial and/or radial position ofthe lenses of the optical elements 1402 and/or the light output from thelight sources 708. Specifically, the motor and lighting module 1406includes a NUV light driver 1534 to transmit a NUV signal to the NUVlight source 708 c, a NIR light driver 1536 to transmit a NIR signal tothe NIR light source 708 b, and a visible light driver 1538 to transmita visible light signal to the visible light source 708 a.

In addition, the motor and lighting module 1406 includes a filter motordriver 1540 to transmit a filter motor signal to a filter motor 1542,which controls the filter 740 of FIGS. 7 and 8. The motor and lightingmodule 1406 includes a rear zoom lens motor driver 1544 to transmit arear zoom lens motor signal to a rear zoom lens motor 1546, a front zoomlens motor driver 1548 to transmit a front zoom lens motor signal to afront zoom lens motor 1550, and a rear working distance lens motordriver 1552 to transmit a working distance lens motor signal to aworking distance lens motor 1554. The motor and lighting module 1406 mayalso include a motor and/or actuator to move and/or tilt the deflectingelement 712.

The rear zoom lens motor 1546 is configured to rotate a drive screw thatcauses carrier 730 to move axially along a track or rail. The front zoomlens motor 1550 is configured to rotate a drive screw that causescarrier 724 to move axially along the track 1106 shown in FIGS. 11 and12. The working distance lens motor 1554 is configured to rotate a drivescrew that causes the rear working distance lens 702 to move axiallyalong a track or rail.

The drivers 1536, 1538, and 1540 may include any type of lightingdriver, transformer, and/or ballast. The drivers 1536, 1538, and 1540are configured to output a pulse width modulation (“PWM”) signal tocontrol an intensity of light output by the light sources 708. In someembodiments, the processor 1522 may control the timing of the drivers1536, 1538, and 1540 to correspond to a timing for applying a certainfilter using the filter motor driver 1540.

The example drivers 1540, 1544, 1548, and 1552 may include, for examplestepper motor drivers and/or DC motor drivers. Likewise, the motors1542, 1546, 1550, and/or 1554 may include a stepper motor, a DC motor,or other electrical, magnetic, thermal, hydraulic, or pneumaticactuator. The motors 1542, 1546, 1550, and/or 1554 may include, forexample, a rotary encoder, a slotted optical switch (e.g., aphotointerrupter), and/or a linear encoder to report an angular positionof a shaft and/or axle for feedback reporting and control. Alternativeembodiments may include voice-coil motors, piezoelectric motors, linearmotors, with suitable drivers, and equivalents thereof.

To control the drivers 1534, 1536, 1538, 1540, 1544, 1548, and 1552, theprocessor 1522 is configured to use a program 1530 for converting acommand message into a digital and/or analog signal. The processor 1522transmits the digital and/or analog signal to the appropriate driver,which outputs an analog power signal, such as a PWM signal correspondingto the received signal. The analog power signal provides power to anappropriate motor or actuator causing it to rotate (or otherwise move)by a desired amount.

The processor 1522 may receive feedback from the drivers 1534, 1536,1538, 1540, 1544, 1548, and 1552, the motors 1542, 1546, 1550, and/or1554, and/or the light sources 708. The feedback corresponds to, forexample, a lighting level or lighting output. Regarding the motors, thefeedback corresponds to a position of a motor (or other actuator) and/oran amount of movement. The processor 1522 uses a program 1530 totranslate the received signal into digital feedback to determine, forexample, a radial, tilt, and/or axial position of a lens based on anangular position of the corresponding motor or actuator shaft. Theprocessor 1522 may then transmit a message with the position informationto the information processor module 1408 for display to a user and/or totrack a position of the lenses of the optical elements 1402 forcalibration.

In some embodiments, the motor and lighting module 1406 may includeadditional drivers to change an axial, tilt, and/or radial position ofindividual lenses within the optical elements 1402. For example, themotor and lighting module 1406 may include drivers that control motorsfor actuating flexures 750 and 752 for the optical image sensors 746 and748 for tilting and/or radial/axial adjustment. Further, the motor andlighting module 1406 may include drivers that control motors (oractuators) for individually tilting and/or adjusting front lenses 720and 722, the front zoom lenses 726 and 728, the rear zoom lenses 732 and734, the lens barrels 736 and 738, and/or final optical elements 745 and747 radially along an x-axis or y-axis and/or axially. Independentadjustment of the lenses and/or sensors enables, for example, the motorand lighting controller 1520 to remove image defects and/or align theleft and right images.

The following sections describe how the processor 1552 executes one ormore programs 1530 to change a working distance, zoom, filter position,lens position, and/or light output.

1. Working Distance Example

The example processor 1522 of the motor and lighting module 1406 of FIG.15 is configured to adjust a working distance of the stereoscopicvisualization camera 300. The working distance is set by adjusting adistance between the rear working distance lens 704 and the frontworking distance lens 408. The processor 1522 adjusts the distance bycausing the rear working distance lens 704 to move relative to the frontworking distance lens 408. Specifically, the processor 1522 sends asignal to the rear working distance lens motor driver 1552, whichactivates the working distance lens motor 1554 for a predetermined timeproportional to an amount the rear working distance lens 704 is to bemoved. The working distance lens motor 1554 drives a leadscrew throughthreads attached to a sliding track that holds the rear working distancelens 704. The working distance lens motor 1554 causes the lens 704 tomove a desired distance, thereby adjusting the working distance. Theworking distance lens motor 1554 may provide a feedback signal to theprocessor 1522, which determines if the rear working distance lens 704was moved the desired amount. If the movement is less or more thandesired, the processor 1522 may send instructions further refining theposition of the rear working distance lens 704. In some embodiments, theinformation processor module 1408 may determine feedback control for therear working distance lens 704.

To determine a position of the rear working distance lens 704, theprocessor 1522 may operate one or more calibration programs 1530. Forexample, upon activation, the processor 1522 may instruct the workingdistance lens motor 1554 to drive a leadscrew to move the rear workingdistance lens 704 along a track or rail until triggering a limit switchat one end of the motion range. The processor 1522 may designate thisstop position as a zero-point for the encoder of the motor 1554. Havingknowledge of the current position of the rear working distance lens 704and the corresponding encoder value, the processor 1522 becomes capableof determining a number of shaft rotations to cause the rear workingdistance lens 704 to move to a desired position. The number of shaftrotations is transmitted in an analog signal to the working distancelens motor 1554 (via the driver 1552) to accordingly move the lens 704to a specified position.

2. Zoom Example

The example processor 1522 of FIG. 15 is configured to execute one ormore programs 1530 to change a zoom level of the stereoscopicvisualization camera 300. As discussed above, zoom (e.g., magnificationchange) is achieved by changing positions of the front zoom set 724 andthe rear zoom set 730 relative to each other and relative to the frontlens set 714 and the lens barrel set 718. Similar to the calibrationprocedure described above for the rear working distance lens 704, theprocessor 1522 may calibrate positions of the sets 724 and 730 alongtracks or rails. Specially, the processor 1522 sends instructionscausing the rear zoom lens motor 1546 and the front zoom lens motor 1550to move the sets 724 and 730 (e.g., carriers) along a rail (or rails) toa stop position at a limit switch. The processor 1522 receives encoderfeedback from the motors 1546 and 1550 to determine an encoder valueassociated with the stop position for the sets 724 and 730. Theprocessor 1522 may then zero-out the encoder value or use the knownencoder value at the stop position to determine how much the motors 1546and 1550 are to be activated to achieve a desired position for the sets724 and 730 along the rail.

In addition to calibration for stop position, the processor 1522 mayexecute programs 1530 that define locations for sets 724 and 730 toachieve a desired zoom level. For example, a known pattern of distancesettings versus a set of desired zoom values may be stored as a program1530 (or a look-up table) during a calibration procedure. Thecalibration procedure may include placing a template within the targetsite 700 and instructing the processor 522 to move the sets 724 and 730until a certain designated marker or character is a certain size inright and left images or frames. For example, a calibration routine maydetermine positions of the set 724 and 730 on a rail corresponding towhen character “E” on a template at the target site 700 is displayed inright and left images as having a height of 10 pixels.

In some embodiments, the information processor module 1408 may performthe visual analysis and send instructions to the processor 1522regarding desired movement for the sets 724 and 730 to zoom in or zoomout. In addition, the information processor 1408 may send instructionsfor moving the focal plane such that the target site 700 at the desiredzoom level is in focus. The instructions may include, for example,instructions to move the rear working distance lens 704 and/or movingthe sets 724 and 730 together and/or individually. In some alternativeembodiments, the processor 1522 may receive calibration parameters forthe rail position of the front zoom set 724 and the rear zoom set 730 atcertain zoom levels from the user input device 1410 or another computer.

The example processor 1522 and/or the information processor module 1408may send instructions such that an image remains in focus whilemagnification changes. The processor 1522, for example, may use aprogram 1530 and/or a look-up-table to determine how certain lenses areto be moved along an optical axis to retain focus on the target site700. The programs 1530 and/or look-up-table may specify magnificationlevels and/or set points on a rail and corresponding lens adjustmentsneeded to keep the focal plane from moving.

Table 2 below shows an example program 1530 or look-up-table that may beused by the processor 1522 to retain focus while changing magnification.The position of the front zoom lens set 724 and the rear zoom lens set730 is normalized based on a length of a rail to stop positions for therespective sets 724 and 730. To decrease magnification, the rear zoomlens set is moved toward the lens barrel set 718, thereby increasing aposition along a rail. The front zoom lens set 724 is also moved.However, its movement does not necessarily equal the movement of therear zoom lens set 730. Instead, the movement of the front zoom lens set724 accounts for changing a distance between the sets 724 and 730 toretain the position of the focal plane to maintain focus while changingmagnifications. For example, to decrease a magnification level from 10×to 9×, the processor 1522 instructs the rear zoom lens set 730 to movefrom position 10 to position 11 along a rail. In addition, the processor1522 instructs the front zoom lens set 724 to move from position 5 toposition 4 along a rail (or same rail as the set 730). Not only have thesets 724 and 730 moved to change magnification, the sets 724 and 730have moved relative to each other to retain focus.

TABLE 2 Front Zoom Lens Rear Zoom Lens Magnification Set Position SetPosition 10X  5 10 9X 4 11 8X 3 12 7X 4.5 14 6X 6 17 5X 8 20

It should be appreciated that Table 2 provides an example of how thesets 724 and 730 may be moved. In other examples, Table 2 may includeadditional rows to account for more precise magnifications and/orpositions of the sets 724 and 730. Additionally or alternatively, Table2 may include a column for the rear working distance lens 704. Forexample, the rear working distance lens 704 may be moved instead of orin conjunction with the front zoom lens set 724 to retain focus.Further, Table 2 may include rows specifying positions for the sets 724and 730 and the rear working distance lens 704 to retain focus duringchanges in working distance.

The values in Table 2 may be determined through calibration and/orreceived from a remote computer or the user input device 1410. Duringcalibration, the information processor module 1408 may operate acalibration program 1560 that progresses through differentmagnifications and/or working distances. A processor 1562 at theinformation processor module 1408 may perform image processing of theimages themselves or received pixel data to determine when a desiredmagnification is achieved using, for example, a template withpredetermined shapes and/or characters. The processor 1562 determines ifthe received images are in-focus. Responsive to determining images areout of focus, the processor 1562 sends instructions to the processor1522 to adjust the front zoom lens set 724 and/or the rear workingdistance lens set 704. The adjustment may include iterative movements inforward and reverse directions along an optical path until the processor1562 determines images are in focus. To determine an image is in focus,the processor 1562 may perform, for example, image analysis searchingfor images where light fuzziness is minimal and/or analyzing pixel datafor differences in light values between adjacent pixel regions (wheregreater differences correspond to more in focus images). Afterdetermining an image is in focus at a desired working distance andmagnification, the processor 1562 and/or the processor 1522 may thenrecord positions of the sets 724 and 730 and/or the rear workingdistance lens 704 and corresponding magnification level.

3. Filter Position Example

The example processor 1522 of the motor and lighting module 1406 of FIG.15 is configured to move the filter 740 into the right and left opticalpaths based on received instructions. In some examples, the filter 740may include a mirror array. In these examples, the processor 1522 sendsinstructions to the filter motor driver 1540 to actuate one or moremotors 1542 to change positions of the mirrors. In some instances, thedriver 1540 may send an electrical charge along one or more paths to thefilter 740, causing certain mirror elements to switch to an on or offposition. In these examples, the filter type selection is generallybinary based on which mirrors to actuate.

In other examples, the filter 740 may include a wheel with differenttypes of filters such as an infrared cut filter, near-infrared bandpassfilter, and near-ultraviolet cut filter. In these examples, the wheel isrotated by the filter motor 1542. The processor 1522 determines stoppositions of the wheel corresponding to partitions between the differentfilters. The processor 1522 also determines rotary encoder valuecorresponding to each of the stop positions.

The processor 1522 may operate a calibration program 1530 and/or theprocessor 1562 may operate a calibration program 1560 to determine thestop positions. For example, the processor 1522 may rotate the filterwheel 740 slowly, with the processor 1562 determining when lightreceived at the pixels changes (using either image analysis or readingpixel data from the image capture module 1404). A change in a lightvalue at the pixels is indicative of a change in the filter type beingapplied to the optical paths). In some instances, the processor 1522 maychange which light sources 708 are activated to create furtherdistinction at the pixels when a different filter type is applied.

4. Light Control and Filter Example

As disclosed above, the processor 1522 may control the light sources 708in conjunction with the filter 740 to cause light of a desiredwavelength to reach the optical image sensors 746 and 748. In someexamples, the processor 1522 may control or synchronize timing betweenactivation of one or more of the light sources 708 and one or more ofthe filters 740. To synchronize timing, a program 1530 may specify adelay time for activating a certain filter. The processor 1522 uses thisprogram 1530 to determine when, for example a signal to activate thefilter 740 is to be transmitted relative to sending a signal to turn ona light source 708. The scheduled timing ensures the appropriate filter740 is applied when the specified light source 708 is activated. Such aconfiguration enables features highlighted by one light source 708 (suchas fluorescence) to be shown on top of or in conjunction with featuresdisplayed under a second light source 708, such as white or ambientlight.

In some instances, the light sources 708 may be switched as fast as thelight filters 740 may be changed, thereby enabling images recorded indifferent lights to be shown in conjunction on top of each other. Forexample, veins or other anatomical structures that emit fluorescence(due to an administered dye or contrast agent) may be shown on top of animage under ambient lighting. In this example, the veins would behighlighted relative to the background anatomical features shown invisible light. In this instance, the processor 1562 and/or a graphicsprocessing unit 1564 (e.g., a video card or graphics card) of theinformation processor module 1408 combines or overlays one or moreimages recorded during application of one filter with images recordedduring application of a subsequent filter.

In some embodiments, the processor 1522 may activate multiple lightsources 708 at the same time. The light sources 708 can be activatedsimultaneously or sequentially to “interleave” light of differentwavelengths to enable different information to be extracted usingappropriate pixels at the optical image sensors 746 and 748. Activatingthe light sources simultaneously may help illuminate dark fields. Forexample, some applications use UV light to stimulate fluorescence at atarget site 700. However, UV light is perceived by an operator as beingvery dark. Accordingly, the processor 1522 may activate the visiblelight source 1538 periodically to add some visible light to the viewingfield so that the surgeon can observe the field-of-view withoutoverwhelming pixels that are sensitive to UV light but can also detectsome visible light. In another example, alternating between lightsources 708 avoids, in some instances, washing out pixels of the opticalimage sensors 746 and 748 that have overlapping sensitivity at the edgesof their ranges.

5. Light Intensity Control

The example processor 1522 of FIG. 15 is configured to execute one ormore programs 1530 to change an intensity of or a level of illuminationprovided by the light sources 708. It should be appreciated that thedepth of field is dependent on the level of illumination at the targetsite 700. Generally, higher illumination provides a greater depth offield. The processor 1522 is configured to ensure an appropriate amountof illumination is provided for a desired depth of field without washingout or overheating the field-of-view.

The visible light source 708 a is driven by the visible light driver1538 and outputs light in the human-visible part of the spectrum as wellas some light outside that region. The NIR light source 708 b is drivenby the NIR light driver 1536 and outputs light primarily at a wavelengththat referred to as near-infrared. The NUV light source 708 c is drivenby the NUV light driver 1534 and outputs light primarily at a wavelengththat is deep in the blue part of the visible spectrum, which is referredto as near-ultraviolet. The respective light drivers 1534, 1536, and1538 are controlled by commands provided by the processor 1522. Controlof the respective output spectra of the light sources 708 is achieved byPWM signal, where a control voltage or current is switched between aminimum (e.g., off) and maximum (e.g., on) value. The brightness of thelight that is output from the light sources 708 is controlled by varyingthe switching rate as well as the percentage of time the voltage orcurrent is at the maximum level per cycle in the PWM signal.

In some examples, the processor 1522 controls an output of the lightsources 708 based on a size of the field-of-view or zoom level. Theprocessor 1522 may execute a program 1530 that specifies for certainlight sensitive settings that light intensity becomes a function ofzoom. The program 1530 may include, for example a look-up-table thatcorrelates a zoom level to a light intensity value. The processor 1522uses the program 1530 to select the PWM signal for the light source 708based on the selected magnification level. In some examples, theprocessor 1522 may reduce light intensity as the magnification increasesto maintain the amount of light provided to the field-of-view per unitof area.

C. Example Information Processor Module

The example information processor module 1408 within the stereoscopicvisualization camera 300 of FIG. 15 is configured to analyze and processimages/frames received from the image capture module 1404 for display.In addition, the information processor module 1408 is configured tointerface with different devices and translate control instructions intomessages for the image capture module 1404 and/or the motor and lightingmodule 1406. The information processor module 1408 may also provide aninterface for manual calibration and/or manage automatic calibration ofthe optical elements 1402.

As shown in FIG. 15, the information processor module 1408 iscommunicatively and/or electrically coupled to the image capture module1404 and the motor and lighting module 1406. For example, thecommunications channel 1514 in addition to communications channels 1566and 1568 may include USB 2.0 or USB 3.0 connections. As such, theinformation processor module 1408 regulates and provides power to themodules 1404 and 1406. In some embodiments, the information processormodule 1408 converts 110-volt alternating current (“AC”) power from awall outlet into a 5, 10, 12, and/or 24 volt direct current (“DC”)supply for the modules 1404 and 1406. Additionally or alternatively, theinformation processor module 1408 receives electrical power from abattery internal to the housing 302 of the stereoscopic visualizationcamera 300 and/or a battery at the cart 510.

The example information processor module 1408 includes thecommunications interface 1532 to communicate bidirectionally with theimage capture module 1404 and the motor and lighting module 1406. Theinformation processor module 1408 also includes the processor 1562configured to execute one or more programs 1560 to process images/framesreceived from the image capture module 1404. The programs 1560 may bestored in a memory 1570. In addition the processor 1562 may performcalibration of the optical elements 1402 and/or adjust the opticalelements 1402 to align right and left images and/or remove visualdefects.

To process images and/or frames into a rendered three-dimensionalstereoscopic display, the example information processor module 1408includes the graphics processing unit 1564. FIG. 16 shows a diagram ofthe graphics processing unit 1564, according to an example embodiment ofthe present disclosure. During operation, the processor 1562 receivesimages and/or frames from the image capture module 1404. An unpackroutine 1602 converts or otherwise changes the images/frames from aformat conducive for transmission across the communications channel 1514into a format conducive for image processing. For instance, the imagesand/or frames may be transmitted across the communications channel 1514in multiple messages. The example unpack routine 1602 combines the datafrom the multiple messages to reassemble the frames/images. In someembodiments, the unpack routine 1602 may queue frames and/or imagesuntil requested by the graphics processing unit 1564. In other examples,the processor 1562 may transmit each right and left image/frame pairafter being completely received and unpacked.

The example graphics processing unit 1564 uses one or more programs 1580(shown in FIG. 15) to prepare images for rendering. Examples of theprograms 1580 are shown in FIGS. 15 and 16. The programs 1580 may beexecuted by a processor of the graphics processing unit 1564.Alternatively, each of the programs 1580 shown in FIG. 16 may beexecuted by a separate graphics processor, microcontroller, and/orapplication specific integrated circuit (“ASIC”). For example, ade-Bayer program 1580 a is configured to smooth or average pixel valuesacross neighboring pixels to compensate for a Bayer pattern applied tothe pixel grids 1002 and 1004 of the right and left optical imagesensors 746 and 748 of FIGS. 7 and 8. The graphics processing unit 1564may also include programs 1580 b, 1580 c, and 1580 d for colorcorrection and/or white balance adjustment. The graphics processing unit1564 also includes a renderer program 1580 e for preparing colorcorrected images/frames for display on the display monitors 512 and 514.The graphics processing unit 1564 may further interact and/or include aperipheral input unit interface 1574, which is configured to combine,fuse, or otherwise include other images and/or graphics for presentationwith the stereoscopic display of the target site 700. Further details ofthe programs 1580 and the information processor module 1408 moregenerally are discussed below.

The example information processor module 1408 may execute one or moreprograms 1562 to check for and improve latency of the stereoscopicvisualization camera 300. Latency refers to the amount of time taken foran event to occur at the target site 700 and for that same event to beshown by the display monitors 512 and 514. Low latency provides afeeling that the stereoscopic visualization camera 300 is an extensionof a surgeon's eyes while high latency tends to distract from themicrosurgical procedure. The example processor 1562 may track how muchtime elapses between images being read from the optical image sensors746 and 748 until the combined stereoscopic image based on the readimages is transmitted for display. Detections of high latency may causethe processor 1562 to reduce queue times, increase the frame rate,and/or skip some color correction steps.

1. User Input Example

The example processor 1562 of the information processor module 1408 ofFIG. 15 is configured to convert user input instructions into messagesfor the motor and lighting module 1406 and/or the image capture module1402. User input instructions may include requests to change opticalaspects of the stereoscopic visualization camera 300 including amagnification level, a working distance, a height of a focal plane(e.g., focus), a lighting source 708, and/or a filter type of the filter740. The user input instructions may also include requests to performcalibration, including indications of an image being in focus and/orindications of image alignment, and/or indications of aligned ZRPsbetween left and right images. The user input instructions may furtherinclude adjustments to parameters of the stereoscopic visualizationcamera 300, such as frame rate, exposure time, color correction, imageresolution, etc.

The user input instructions may be received from a user input device1410, which may include the controls 305 of the control arm 304 of FIG.3 and/or a remote control. The user input device 1410 may also include acomputer, tablet computer, etc. In some embodiments, the instructionsare received via a network interface 1572 and/or a peripheral input unitinterface 1574. In other embodiments, the instructions may be receivedfrom a wired connection and/or a RF interface.

The example processor 1562 includes programs 1560 for determining aninstruction type and determining how the user input is to be processed.In an example, a user may press a button of the control 305 to change amagnification level. The button may continue to be pressed until theoperator has caused the stereoscopic visualization camera 300 to reach adesired magnification level. In these examples, the user inputinstructions include information indicative that a magnification levelis to be, for example, increased. For each instruction received (or eachtime period in which a signal indicative of the instruction isreceived), the processor 1562 sends a control instruction to the motorand lighting processor 1406 indicative of the change in magnification.The processor 1522 determines from a program 1530 how much the zoom lenssets 724 and 730 are to be moved using, for example, Table 2. Theprocessor 1522 accordingly transmits a signal or message to the rearzoom lens motor driver 1544 and/or the front zoom lens motor driver 1548causing the rear zoom lens motor 1546 and/or the front zoom lens motor1550 to move the rear zoom lens set 730 and/or the front zoom lens set724 by an amount specified by the processor 1562 to achieve the desiredmagnification level.

It should be appreciated that in the above example, the stereoscopicvisualization camera 300 provides a change based on user input but alsomakes automatic adjustments to maintain focus and/or a high imagequality. For instance, instead of simply changing the magnificationlevel, the processor 1522 determines how the zoom lens sets 724 and 730are to be moved to also retain focus, thereby saving an operator fromhaving to perform this task manually. In addition, the processor 1562may, in real-time, adjust and/or align ZRPs within the right and leftimages as a magnification level changes. This may be done, for example,by selecting or changing locations of the pixel sets 1006 and 1008 withrespect to pixel grids 1002 and 1004 of FIG. 10.

In another example, the processor 1562 may receive an instruction fromthe user input device 1410 to change a frame rate. The processor 1562transmits a message to the processor 1504 of the image capture module1404. In turn, the processor 1504 writes to registers of the right andleft image sensors 746 and 748 indicative of the new frame rate. Theprocessor 1504 may also update internal registers with the new framerate to change a pace at which the pixels are read.

In yet another example, the processor 1562 may receive an instructionfrom the user input device 1410 to begin a calibration routine for ZRP.In response, the processor 1562 may execute a program 1560 thatspecifies how the calibration is to be operated. The program 1560 mayinclude, for example, a progression or iteration of magnification levelsand/or working distances in addition to a routine for verifying imagequality. The routine may specify that for each magnification level,focus is to be verified in addition to ZRP. The routine may also specifyhow the zoom lens sets 724 and 730 and/or the rear working distance lens704 are to be adjusted to achieve an in focus image. The routine mayfurther specify how ZRP of the right and left images are to be centeredfor the magnification level. The program 1560 may store (to alook-up-table) locations of zoom lens sets 724 and/or the 730 and/or therear working distance lens 704 in addition to locations of pixel sets1006 and 1008 and the corresponding magnification level once imagequality has been verified. Thus, when the same magnification level isrequested at a subsequent time, the processor 1562 uses thelook-up-table to specify positions for the zoom lens sets 724 and/or the730 and/or the rear working distance lens 704 to the motor and lightingmodule 1406 and positions for the pixel sets 1006 and 1008 to the imagecapture module 1404. It should be appreciated that in some calibrationroutines, at least some of the lenses of the optical elements 1402 maybe adjusted radially/rotationally and/or tilted to center ZRPs and/oralign right and left images.

2. Interface Example

To facilitate communications between the stereoscopic visualizationcamera 300 and external devices, the example information processormodule 1408 includes the network interface 1572 and the peripheral inputunit interface 1574. The example network interface 1572 is configured toenable remote devices to communicatively couple to the informationprocessor module 1408 to, for example, store recorded video, control aworking distance, zoom level, focus, calibration, or other features ofthe stereoscopic visualization camera 300. In some embodiments, theremote devices may provide values or parameters for calibrationlook-up-tables or more generally, programs 1530 with calibratedparameters. The network interface 1572 may include an Ethernetinterface, a local area network interface, and/or a Wi-Fi interface.

The example peripheral input unit interface 1574 is configured tocommunicatively couple to one or more peripheral devices 1576 andfacilitate the integration of stereoscopic image data with peripheraldata, such as patient physiological data. The peripheral input unitinterface 1574 may include a Bluetooth® interface, a USB interface, anHDMI interface, SDI, etc. In some embodiments, the peripheral input unitinterface 1574 may be combined with the network interface 1572.

The peripheral devices 1576 may include, for example, data or videostorage units, patient physiological sensors, medical imaging devices,infusion pumps, dialysis machines, and/or tablet computers, etc. Theperipheral data may include image data from a dedicated two-dimensionalinfrared-specialized camera, diagnostic images from a user's laptopcomputer, and/or images or patient diagnostic text from an ophthalmicdevice such as the Alcon Constellation® system and the WaveTec OptiwaveRefractive Analysis (ORA™) system.

The example peripheral input unit interface 1574 is configured toconvert and/or format data from the peripheral devices 1576 into anappropriate digital form for use with stereoscopic images. Once indigital form, the graphics processing unit 1564 integrates theperipheral data with other system data and/or the stereoscopicimages/frames. The data is rendered with the stereoscopic images fordisplay on the display monitors 512 and/or 514.

To configure the inclusion of peripheral data with the stereoscopicimages, the processor 1562 may control an integration setup. In anexample, the processor 1562 may cause the graphics processing unit 1564to display a configuration panel on the display monitors 512 and/or 514.The configuration panel may enable an operator to connect a peripheraldevice 1576 to the interface 1574 and the processor 1562 to subsequentlyestablish communications with the device 1576. The processor 1564 maythen read which data is available or enable the operator to use theconfiguration panel to select a data directory location. Peripheral datain the directory location is displayed in the configuration panel. Theconfiguration panel may also provide the operator an option to overlaythe peripheral data with stereoscopic image data or display as aseparate picture.

Selection of peripheral data (and overlay format) causes the processor1562 to read and transmit the data to the graphics processing unit 1564.The graphics processing unit 1564 applies the peripheral data to thestereoscopic image data for presentation as an overlay graphic (such asfusing a preoperative image or graphic with a real-time stereoscopicimage), a “picture-in-picture,” and/or a sub-window to the side or ontop of the main stereoscopic image window.

3. De-Bayer Program Example

The example de-Bayer program 1580 a of FIG. 16 is configured to produceimages and/or frames with values for red, green, and blue color at everypixel value. As discussed above, the pixels of the right and leftoptical image sensors 746 and 748 have a filter that passes light in thered wavelength range, the blue wavelength range, or the green wavelengthrange. Thus, each pixel only contains a portion of the light data.Accordingly, each image and/or frame received in the informationprocessor module 1408 from the image capture module 1404 has pixels thatcontain either red, blue, or green pixel data.

The example de-Bayer program 1580 a is configured to average the red,blue, and green pixel data of adjacent and/or neighboring pixels todetermine more complete color data for each pixel. In an example, apixel with red data and a pixel with blue data are located between twopixels with green data. The green pixel data for the two pixels isaveraged and assigned to the pixel with red data and the pixel with bluedata. In some instances, the averaged green data may be weighted basedon a distance of the pixel with red data and the pixel with blue datafrom the respective green pixels. After the calculation, the pixels withoriginally only red or blue data now include green data. Thus, after thede-Bayer program 1580 a is executed by the graphics processing unit1564, each pixel contains pixel data for an amount of red, blue, andgreen light. The pixel data for the different colors is blended todetermine a resulting color on the color spectrum, which may be used bythe renderer program 1580 e for display and/or the display monitors 512and 514. In some examples, the de-Bayer program 1580 a may determine theresulting color and store data or an identifier indicative of the color.

4. Color Correction Example

The example color correction programs 1580 b, 1580 c, and 1580 d areconfigured to adjust pixel color data. The sensor color correctionprogram 1580 b is configured to account or adjust for variability incolor sensing of the optical image sensors 746 and 748. The user colorcorrection program 1580 c is configured to adjust pixel color data basedon perceptions and feedback of an operator. Further, the display colorcorrection program 1580 d is configured to adjust pixel color data basedon a display monitor type.

To correct color for sensor variability, the example color correctionprogram 1580 b specifies a calibration routine that is executable by thegraphics processing unit 1564 and/or the processor 1562. The sensorcalibration includes placing a calibrated color chart, such as theColorChecker® Digital SG by X-Rite, Inc. at the target site 700. Theprocessor 1562 and/or the graphics processing unit 1564 executes theprogram 1580 b, which includes sending instructions to the image capturemodule 1404 to record right and left images of the color chart. Pixeldata from the right and left images (after being processed by thede-Bayer program 1580 a) may be compared to pixel data associated withthe color chart, which may be stored to the memory 1570 from aperipheral unit 1576 and/or a remote computer via the network interface1572. The processor 1562 and/or the graphics processing unit 1564determines differences between the pixel data. The differences arestored to the memory 1570 as calibration data or parameters. The sensorcolor correction program 1580 b applies the calibration parameters tosubsequent right and left images.

In some examples, the differences may be averaged over regions of pixelssuch that the program 1580 b finds a best-fit of color correction datathat can be applied globally to all of the pixels of the optical imagessensors 746 and 748 to produce colors as close to the color chart aspossible. Additionally or alternatively, the program 1580 b may processuser input instructions received from the user unit device 1410 tocorrect colors. The instructions may include regional and/or globalchanges to red, blue, and green pixel data based on operatorpreferences.

The example sensor color correction program 1580 b is also configured tocorrect for white balance. Generally, white light should result in red,green, and blue pixels having equal values. However, differences betweenpixels can result from color temperature of light used during imaging,inherent aspects of the filter and sensing element of each of thepixels, and spectral filtering parameters of, for example, thedeflecting element 712 of FIGS. 7 and 8. The example sensor colorcorrection program 1580 b is configured to specify a calibration routineto correct for the light imbalances.

To perform white balance, the processor 1562 (per instructions from theprogram 1580 b) may display an instruction on the display monitor 512and/or 514 for an operator to place a neutral card at the target site700. The processor 1562 may then instruct the image capture module 1404to record one or more images of the neutral card. After processing bythe unpack routine 1602 and the de-Bayer program 1580 a, the program1580 b determines regional and/or global white balance calibrationweight values for each of the red, blue, and green data such that eachof the pixels have substantially equal values of red, blue, and greendata. The white balance calibration weight values are stored to thememory 1570. During operation, the graphics processing unit 1564 usesthe program 1580 b to apply the white balance calibration parameters toprovide white balance.

In some examples, the program 1580 b determines white balancecalibration parameters individually for the right and left optical imagesensors 746 and 748. Of these examples, the program 1580 b may storeseparate calibration parameters for the left and right images. In otherinstances, the sensor color correction program 1580 b determines aweighting between the right and left views such that color pixel data isnearly identical for the right and left optical image sensors 746 and748. The determined weight may be applied to the white balancecalibration parameters for subsequent use during operation of thestereoscopic visualization camera 300.

In some embodiments, the sensor color correction program 1580 b of FIG.16 specifies that the white balance calibration parameters are to beapplied as a digital gain on the pixels of the right and left opticalimage sensors 746 and 748. For example, the processor 1504 of the imagecapture module 1404 applies the digital gain to pixel data read fromeach of the pixels. In other embodiments, the white balance calibrationparameters are to be applied as an analog gain for each pixel's colorsensing element.

The example sensor color correction program 1580 b may perform whitebalancing and/or color correction when the different light sources 708and/or filter types of the filter 740 are activated. As a result, thememory 1570 may store different calibration parameters based on whichlight source 708 is selected. Further, the sensor color correctionprogram 1580 b may perform white balancing and/or color correction fordifferent types of external light. An operator may use the user inputdevice 1410 to specify characteristics and/or a type of the externallight source. This calibration enables the stereoscopic visualizationcamera 300 to provide color correction and/or white balance fordifferent lighting environments.

The example program 1580 b is configured to perform calibration on eachof the optical image sensors 746 and 748 separately. Accordingly, theprogram 1580 b applies different calibration parameters to the right andleft images during operation. However, in some examples, calibration mayonly be performed on one sensor 746 or 748 with the calibrationparameters being used for the other sensor.

The example user color correction program 1580 c is configured torequest operator-provided feedback regarding image quality parameterssuch as brightness, contrast, gamma, hue, and/or saturation. Thefeedback may be received as instructions from the user input device1410. Adjustments made by the user are stored as user calibrationparameters in the memory 1570. These parameters are subsequently appliedby the user color correction program 1580 c to right and left opticalimages after color correction for the optical image sensors 746 and 748.

The example display color correction program 1580 d of FIG. 16 isconfigured to correct image color for a display monitor using, forexample, the Datacolor™ Spyder color checker. The program 1580 d,similar to the program 1580 b, instructs the image capture module 1404to record an image of a display color template at the target scene 700.The display color correction program 1580 d operates a routine to adjustpixel data to match an expected display output stored in a look-up-tablein the memory 1570. The adjusted pixel data may be stored as displaycalibration parameters to the memory 1570. In some examples, a camera orother imaging sensor may be connected to the peripheral input unitinterface 1574, which provides images or other feedback regarding colorrecorded from the display monitors 512 and 514, which is used to adjustthe pixel data.

5. Stereoscopic Image Display Example

The example renderer program 1580 e of the graphics processing unit 1564of FIG. 16 is configured to prepare right and left images and/or framesfor three-dimensional stereoscopic display. After the pixel data of theright and left images is color corrected by the programs 1580 b, 1580 c,and 1580 d, the renderer program 1580 e is configured to draw left-eyeand right-eye data into a format suitable for stereoscopic display andplace the final rendered version into an output buffer for transmissionto one of the display monitors 512 or 514.

Generally, the renderer program 1580 e receives a right image and/orframe and a left image and/or frame. The renderer program 1580 ecombines the right and left images and/or frames into a single frame. Insome embodiments, the program 1580 e operates a top-bottom mode andcondenses the left image data in height by half. The program 1580 e thenplaces the condensed left image data in a top half of the combinedframe. Similarly, the program 1580 e condenses the right image data inheight by half and places the condensed right image data in a bottomhalf of the combined frame.

In other embodiments, the renderer program 1580 e operates aside-by-side mode where each of the left and right images are condensedin width by half and combined in a single image such that the left imagedata is provided on a left half of the image while right image data isprovided on a right half of the image. In yet an alternative embodiment,the renderer program 1580 e operates a row-interleaved mode where everyother line in the left and right frames is discarded. The left and rightframes are combined together to form a complete stereoscopic image.

The example renderer program 1580 e is configured to render combinedleft and right images separately for each connected display monitor. Forinstance, if both the display monitors 512 and 514 are connected, therenderer program 1580 e renders a first combined stereoscopic image forthe display monitor 512 and a second combined stereoscopic image for thedisplay monitor 514. The renderer program 1580 e formats the first andsecond combined stereoscopic images such that they are compatible withthe type and/or screen size of the display monitors and/or screen.

In some embodiments, the renderer program 1580 e selects the imageprocessing mode based on how the display monitor is to displaystereoscopic data. Proper interpretation of stereoscopic image data bythe brain of an operator requires that the left eye data of thestereoscopic image be conveyed to the operator's left eye and the righteye data of the stereoscopic image be conveyed to the operator's righteye. Generally, display monitors provide a first polarization for lefteye data and a second opposing polarization for the right eye data.Thus, the combined stereoscopic image must match the polarization of thedisplay monitor.

FIG. 17 shows an example of the display monitor 512, according to anexample embodiment of the present disclosure. The display monitor 512may be, for example, the LG® 55LW5600 three-dimensional television witha screen 1702. The example display monitor 512 uses a polarization filmon the screen 1702 such that all odd rows 1704 have a first polarizationand all even rows 1706 have an opposing polarization. For compatibilitywith the display monitor 512 shown in FIG. 17, the renderer program 1580e would have to select the row-interleaved mode such that the left andright image data are on alternating lines. In some instances, therenderer program 1580 e may request (or otherwise receive) displaycharacteristics of the display monitor 512 prior to preparing thestereoscopic image.

To view the stereoscopic image displayed on the screen 1702, the surgeon504 (remember him from FIG. 5) wears glasses 1712 that include a leftlens 1714 that comprises a first polarization that matches the firstpolarization of the rows 1704. In addition, the glasses 1712 include aright lens 1716 that comprises a second polarization that matches thesecond polarization of the rows 1706. Thus, the left lens 1714 onlypermits a majority of the light from the left image data from the leftrows 1704 to pass through while blocking a majority of the light fromthe right image data. In addition, the right lens 1716 permits amajority of the light from the right image data from the right rows 1706to pass through while blocking a majority of the light from the leftimage data. The amount of light from the “wrong” view that reaches eachrespective eye is known as “crosstalk” and is generally held to a valuelow enough to permit comfortable viewing. Accordingly, the surgeon 504views left image data recorded by the left optical image sensor 748 in aleft eye while viewing right image data recorded by the right opticalimage sensor 746 in a right eye. The surgeon's brain fuses the two viewstogether to create a perception of three-dimensional distance and/ordepth. Further, the use of such a display monitor is advantageous forobserving the accuracy of the stereoscopic visualization camera 300. Ifthe surgeon or operator does not wear glasses, then both left and rightviews are observable with both eyes. If a planar target is placed at thefocal plane, the two images will be theoretically aligned. Ifmisalignment is detected, a re-calibration procedure can be initiated bythe processor 1562.

The example renderer program 1580 e is configured to render the left andright views for circular polarization. However, in other embodiments,the renderer program 1580 e may provide a stereoscopic image compatiblewith linear polarization. Regardless of which type of polarization isused, the example processor 1562 may execute a program 1560 to verify orcheck a polarity of the stereoscopic images being output by the rendererprogram 1580 e. To check polarity, the processor 1562 and/or theperipheral input unit interface 1574 inserts diagnostic data into theleft and/or right images. For example, the processor 1562 and/or theperipheral input unit interface 1574 may overlay “left” text onto theleft image and “right” text onto the right image. The processor 1562and/or the peripheral input unit interface 1574 may display a promptinstructing an operator to close one eye at a time while wearing theglasses 1712 to confirm the left view is being received at the left eyeand the right view is being received at the right eye. The operator mayprovide confirmation via the user input device 1410 indicating whetherthe polarization is correct. If the polarization is not correct, theexample renderer program 1580 e is configured to reverse locations wherethe left and right images are inserted into the combined stereoscopicimage.

In yet other embodiments, the example renderer program 1580 e isconfigured to provide for frame sequential projection instead ofcreating a combined stereoscopic image. Here, the renderer program 1580e renders the left images and or frames time-sequentially interleavedwith the right images and/or frames. Accordingly the left and rightimages are alternately presented to the surgeon 504. In these otherembodiments, the screen 1702 is not polarized. Instead, the left andright lenses of the glasses 1712 may be electronically or opticallysynchronized to their respective portion of a frame sequence, whichprovides corresponding left and right views to a user to discern depth.

In some examples, the renderer program 1580 e may provide certain of theright and left images for display on separate display monitors orseparate windows on one display monitor. Such a configuration may beespecially beneficial when lenses of right and left optical paths of theoptical elements 1402 are independently adjustable. In an example, aright optical path may be set a first magnification level while a leftoptical path is set at a second magnification level. The examplerenderer program 1580 e may accordingly display a stream of images fromthe left view on the display monitor 512 and a stream of images from theright view on the display monitor 514. In some instances, the left viewmay be displayed in a first window on the display monitor 512 while theright view is displayed in a second window (e.g., a picture-in-picture)of the same display monitor 512. Thus, while not stereoscopic, theconcurrent display of the left and right images provides usefulinformation to a surgeon.

In another example, the light sources 708 and the filter 740 may beswitched quickly to generate alternating images with visible light andfluorescent light. The example renderer program 1580 e may combine theleft and right views to provide a stereoscopic display under differentlighting sources to highlight, for example, a vein with a dye agentwhile showing the background in visible light.

In yet another example, a digital zoom may be applied to the rightand/or left optical image sensor 746 or 748. Digital zoom generallyaffects the perceived resolution of the image and is dependent onfactors such as the display resolution and the preference of the viewer.For example, the processor 1504 of the image capture module 1404 mayapply digital zooming by creating interpolated pixels synthesized andinterspersed between the digitally-zoomed pixels. The processor 1504 mayoperate a program 1510 that coordinates the selection and interpolationpixels for the optical image sensors 746 and 748. The processor 1504transmits the right and left images with digital zoom applied to theinformation processor module 1408 for subsequent rendering and display.

In some embodiments, the processor 1504 receives instructions from theprocessor 1562 that a digital zoom image is to be recorded betweenimages without digital zoom to provide a picture-in-picture (or separatewindow) display of a digital zoom of a region of interest of the targetsite 700. The processor 1504 accordingly applies digital zooming toevery other read from the pixel grids 1002 and 1004. This enables therenderer program 1580 e to display simultaneously a stereoscopic fullresolution image in addition to a digitally-zoomed stereoscopic image.Alternatively, the image to be zoomed digitally is copied from thecurrent image, scaled, and placed during the render phase in the properposition overlaid atop the current image. This alternativelyconfiguration avoids the “alternating” recording requirement.

6. Calibration Example

The example information processor module 1408 of FIGS. 14 to 16 may beconfigured to execute one or more calibration programs 1560 tocalibrate, for example, a working distance and/or magnification. Forexample, the processor 1562 may send instructions to the motor andlighting module 1406 to perform a calibration step for mapping a workingdistance (measured in millimeters) from the main objective assembly 702to the target site 700 to a known motor position of the working distancelens motor 1554. The processor 1562 performs the calibration bysequentially moving an object plane in discrete steps along the opticalaxis and re-focusing the left and right images, while recording encodercounts and the working distance. In some examples, the working distancemay be measured by an external device, which transmits the measuredworking distance values to the processor 1562 via the peripheral inputunit interface 1574 and/or an interface to the user input device 1410.The processor 1562 may store the position of the rear working distancelens 704 (based on position of the working distance lens motor 1554) andthe corresponding working distance.

The example processor 1562 may also execute a program 1560 to performmagnification calibration. The processor 1562 may set the opticalelements 1402, using the motor and lighting module 1406 to selectmagnification levels. The processor 1562 may record positions of theoptical elements 1402, or corresponding motor positions with respect toeach magnification level. The magnification level may be determined bymeasuring a height in an image of an object of a known size. Forexample, the processor 1562 may measure an object as having a height of10 pixels and use a look-up-table to determine that a 10 pixel heightcorresponds to a 5× magnification.

To match the stereoscopic perspectives of two different imagingmodalities it is often desirable to model them both as if they aresimple pinhole cameras. The perspective of a 3D computer model, such asa Mill brain tumor, can be viewed from user-adjustable directions anddistances (e.g. as if the images are recorded by a synthesizedstereoscopic camera). The adjustability can be used to match theperspective of the live surgical image, which must therefore be known.The example processor 1562 may calibrate one or more of these pinholecamera model parameters such as, for example, a center of projection(“COP”) of the right and left optical image sensors 746 and 748. Todetermine center of projection, the processor 1562 determines a focusdistance from the center of projection to an object plane. First, theprocessor 1562 sets the optical elements 1402 at a magnification level.The processor 1562 then records measurements of a height of an image atthree different distances along the optical axis including at the objectplane, a distance d less than the object plane distance, and a distanced greater than the object plane distance. The processor 1562 uses analgebraic formula for similar triangles at the two most extremepositions to determine the focus distance to the center of projection.The processor 1562 may determine focus distances at other magnificationsusing the same method or by determining a ratio between themagnifications used for calibration. The processor may use a center ofprojection to match the perspective of an image of a desired fusionobject, such as an Mill tumor model, to a live stereoscopic surgicalimage. Additionally or alternatively, existing camera calibrationprocedures such as OpenCV calibrateCamera may be used to find theabove-described parameters as well as additional camera information suchas a distortion model for the optical elements 1402.

The example processor 1562 may further calibrate the left and rightoptical axes. The processor 1562 determines an interpupillary distancebetween the left and right optical axes for calibration. To determinethe interpupillary distance, the example processor 1562 records left andright images where pixel sets 1006 and 1008 are centered at the pixelgrids 1002 and 1004. The processor 1562 determines locations of ZRPs(and/or distances to a displaced object) for the left and right images,which are indicative of image misalignment and degree of parallax. Inaddition, the processor 1562 scales the parallax and/or the distancebased on the magnification level. The processor 1562 then determines theinterpupillary distance using a triangulation calculation taking intoaccount the degree of parallax and/or the scaled distance to the objectin the display. The processor 1562 next associates the interpupillarydistance with the optical axis at the specified magnification level as acalibration point.

VI. Image Alignment and Spurious Parallax Adjustment Embodiment

Similar to human vision, stereoscopic images comprise right views andleft views that converge at a point of interest. The right and leftviews are recorded at slightly different angles from the point ofinterest, which results in parallax between the two views. Items in thescene in front of or behind the point of interest exhibit parallax suchthat distance or depth of the items from the viewer can be deduced. Theaccuracy of the perceived distance is dependent on, for example, theclarity of the viewer's eyesight. Most humans exhibit some level ofimperfection in their eyesight, resulting in some inaccuracies betweenthe right and left views. However, they are still able to achievestereopsis, with the brain fusing the views with some level of accuracy.

When left and right images are recorded by a camera instead of beingviewed by a human, the parallax between the combined images on a displayscreen produces stereopsis, which provides an appearance of athree-dimensional stereoscopic image on a two-dimensional display.Errors in the parallax can affect the quality of the three-dimensionalstereoscopic image. The inaccuracy of the observed parallax incomparison to a theoretically perfect parallax is known as spuriousparallax. Unlike humans, cameras do not have brains that automaticallycompensate for the inaccuracies.

If spurious parallax becomes significant, the three-dimensionalstereoscopic image may be unviewable to the point of inducing vertigo,headaches, and nausea. There are many factors that can affect theparallax in a microscope and/or camera. For instance, optical channelsof the right and left views may not be exactly equal. The opticalchannels may have unmatched focus, magnification, and/or misalignment ofpoints of interest. These issues may have varying severity at differentmagnifications and/or working distances, thereby reducing efforts tocorrect through calibration.

Known surgical microscopes, such as the surgical microscope 200 of FIG.2 are configured to provide an adequate view through the oculars 206.Often, the image quality of optical elements of known surgicalmicroscopes is not sufficient for stereoscopic cameras. The reason forthis is because manufacturers of surgical microscopes assume the primaryviewing is through oculars. Any camera attachment (such as the camera212) is either monoscopic and not subject to spurious parallax orstereoscopic with low image resolution where spurious parallax is not asapparent.

International standards, such as ISO 10936-1:2000, Optics and opticalinstruments—Operation microscopes—Part 1: Requirements and test methods,have been developed to provide specification limits for image quality ofsurgical microscopes. The specification limits are generally set forviewing through the oculars of a surgical microscope and do not considerthree-dimensional stereoscopic display. For example, regarding spuriousparallax, ISO 10936-1:2000 specifies that the difference in verticalaxes between the left and right views should be less than 15arc-minutes. Small angular deviations of axes are often quantified inarc-minutes, which corresponds to 1/60^(th) of a degree, or arc-seconds,which corresponds to 1/60^(th) of an arc-minute. The 15 arc-minutespecification limit corresponds to a 3% difference between left andright views for a typical surgical microscope with a working distance of250 mm and a field-of-view of 35 mm (which has an angular field-of-viewof 8°).

The 3% difference is acceptable for ocular viewing where a surgeon'sbrain is able to overcome the small degree of error. However, this 3%difference produces noticeable differences between left and right viewswhen viewed stereoscopically on a display monitor. For example, when theleft and right views are shown together, a 3% difference results in animage that appears disjointed and difficult to view for extended periodsof time.

Another issue is that known surgical microscopes may satisfy the 15arc-minute specification limit at only one or a few magnification levelsand/or only individual optical elements may satisfy a certainspecification limit. For example, individual lenses are manufactured tomeet certain criteria. However, when the individual optical elements arecombined in an optical path, small deviations from the standard may beamplified rather than canceled. This can be especially pronounced whenfive or more optical elements are used in an optical path including acommon main objective lens. In addition, it is very difficult toperfectly match optical elements on parallel channels. At most, duringmanufacture, the optical elements of a surgical microscope arecalibrated only at one or a few certain magnification levels to meet the15 arc-minute specification limit. Accordingly, the error may be greaterbetween the calibration points despite the surgical microscope allegedlymeeting the ISO 10936-1:2000 specifications.

In addition, the ISO 10936-1:2000 specification permits largertolerances when additional components are added. For example, addingsecond oculars (e.g., the oculars 208) increases the spurious parallaxby 2 arc-minutes. Again, while this error may be acceptable for viewingthrough oculars 206 and 208, image misalignment becomes more pronouncedwhen viewed stereoscopically through the camera.

In comparison to known surgical microscopes, the example stereoscopicvisualization camera 300 disclosed herein is configured to automaticallyadjust at least some of the optical elements 1402 to reduce or eliminatespurious parallax. Embedding the optical elements within thestereoscopic visualization camera 300 enables fine adjustments to bemade automatically (sometimes in real-time) for three-dimensionalstereoscopic display. In some embodiments, the example stereoscopicvisualization camera 300 may provide an accuracy of 20 to 40arc-seconds, which is close to a 97% reduction in optical error comparedto the 15 arc-minute accuracy of known surgical microscopes.

The improvement in accuracy enables the example stereoscopicvisualization camera 300 to provide features that are not capable ofbeing performed with known stereoscopic microscopes. For example, manynew microsurgical procedures rely on accurate measurements in a livesurgical site for optimal sizing, positioning, matching, directing, anddiagnosing. This includes determining a size of a vessel, an angle ofplacement of a toric Intra Ocular Lens (“IOL”), a matching ofvasculature from a pre-operative image to a live view, a depth of atumor below an artery, etc. The example stereoscopic visualizationcamera 300 accordingly enables precise measurements to be made using,for example, graphical overlays or image analysis to determine sizes ofanatomical structures.

Known surgical microscopes require that a surgeon place an object of aknown size (such as a micro-ruler) into the field-of-view. The surgeoncompares the size of the object to surrounding anatomical structure todetermine an approximate size. However, this procedure is relativelyslow since the surgeon has to place the object in the proper location,and then remove it after the measurement is performed. In addition, themeasurement only provides an approximation since the size is based onthe surgeon's subjective comparison and measurement. Some knownstereoscopic cameras provide graphical overlays to determine size.However, the accuracy of these overlays is reduced if spurious parallaxexists between the left and right views.

A. ZRP as a Source of Spurious Parallax

ZRP inaccuracy provides a significant source of error between left andright images resulting in spurious parallax. ZRP, or zoom repeat point,refers to a point in a field-of-view that remains in a same location asa magnification level is changed. FIGS. 18 and 19 show examples of ZRPin a left and right field-of-view for different magnification levels.Specifically, FIG. 18 shows a left field-of-view 1800 for a lowmagnification level and a left field-of-view 1850 for a highmagnification level In addition, FIG. 19 shows a right field-of-view1900 for a low magnification level and a right field-of-view 1950 for ahigh magnification level.

It should be noted that FIGS. 18 and 19 show crosshairs 1802 and 1902 toprovide an exemplary point of reference for this disclosure. Thecrosshairs 1802 include a first crosshair 1802 a positioned along ay-direction or y-axis and a second crosshair 1802 b positioned along anx-direction or x-axis. Additionally, crosshairs 1902 include a firstcrosshair 1902 a positioned along a y-direction or y-axis and a secondcrosshair 1902 b positioned along an x-direction or x-axis In actualimplementation, the example stereoscopic visualization camera 300 bydefault typically does not include or add crosshairs to the optical pathunless requested by an operator.

Ideally, the ZRP should be positioned at a central location or originpoint. For example, the ZRP should be centered in the crosshairs 1802and 1902. However, inaccuracies in the optical elements 1402 and/orslight misalignments between the optical elements 1402 cause the ZRP tobe located away from the center of the crosshairs 1802 and 1902. Thedegree of spurious parallax corresponds to how far each of the ZRPs ofthe left and right views is located away from the respective centers inaddition to ZRPs being misaligned between the left and right views.Moreover, inaccuracies in the optical elements 1402 may cause the ZRP todrift slightly as magnification changes, thereby further causing agreater degree of spurious parallax.

FIG. 18 shows three crescent-shaped objects 1804, 1806, and 1808 in thefield-of-views 1800 and 1850 of the target site 700 of FIG. 7. It shouldbe appreciated that the field-of-views 1800 and 1850 are linearfield-of-views with respect to the optical image sensors 746 and 748.The objects 1804, 1806, and 1808 were placed in the field-of-view 1800to illustrate how spurious parallax is generated from left and rightimage misalignment. The object 1804 is positioned above crosshair 1802 balong crosshair 1802 a. The object 1806 is positioned along crosshair1802 b and to the left of the crosshair 1802 a. The object 1808 ispositioned slightly below the crosshair 1802 b and to the right of thecrosshair 1802 a. A ZRP 1810 for the left field-of-view 1800 ispositioned in a notch of the object 1808.

The left field-of-view 1800 is changed to the left field-of-view 1850 byincreasing the magnification level (e.g., zooming) using the zoom lensassembly 716 of the example stereoscopic visualization camera 300.Increasing the magnification causes the objects 1804, 1806, and 1808 toappear to expand or grow, as shown in the field-of-view 1850. In theillustrated example, the field-of-view 1850 is approximately 3× themagnification level of the field-of-view 1800.

Compared to the low magnification field-of-view 1800, the objects 1804,1806, and 1808 in high magnification field-of-view 1850 have increasedin size by about 3× while also moving apart from each other by 3× withrespect to the ZRP 1810. In addition, the positions of the objects 1804,1806, and 1808 have moved relative to the crosshairs 1802. The object1804 is now shifted to the left of the crosshair 1802 a and shiftedslightly further from the crosshair 1802 b. In addition, the object 1806is now shifted further to the left of crosshair 1802 a and slightlyabove the crosshair 1802 b. Generally, the object 1808 is located in thesame (or nearly the same) position with respect to the crosshairs 1802,with the ZRP 1810 being located in the exact same (or nearly the same)position with respect to the crosshairs 1802 and the object 1806. Inother words, as magnification increases, the objects 1804, 1806, and1808 (and anything else in the field-of-view 1850) appear to move awayand outward from the ZRP 1810.

The same objects 1804, 1806, and 1808 are shown in the rightfield-of-views 1900 and 1950 illustrated in FIG. 19. However, thelocation of the ZRP is different. Specifically, ZRP 1910 is locatedabove crosshair 1902 b and to the left of crosshair 1902 a in the rightfield-of-views 1900 and 1950. Thus, the ZRP 1910 is located at adifferent location than the ZRP 1810 in the left field-of-views 1800 and1850. In the illustrated example, it is assumed that the left and rightoptical paths are perfectly aligned at the first magnification level.Accordingly, the objects 1804, 1806, and 1808 shown in the rightfield-of-view 1900 in the same location as the same objects 1804, 1806,and 1808 in the left field-of-view 1800. Since the left and right viewsare aligned, no spurious parallax exists.

However, in the high magnification field-of-view 1950, the objects 1804,1806, and 1808 expand and move away from the ZRP 1910. Given thelocation of the ZRP 1910, the object 1804 moves or shifts to the rightand the object 1806 moves or shifts downward. In addition, the object1808 moves downward and to the right compared to its location in thefield-of-view 1900.

FIG. 20 shows a pixel diagram comparing the high magnification leftfield-of-view 1850 to the high magnification right field-of-view. A grid2000 may represent locations of the objects 1804(L), 1806(L), and1808(L) on the pixel grid 1004 of the left optical image sensor 748overlaid with locations of the objects 1804(R), 1806(R), and 1808(R) onthe pixel grid 1002 of the left optical image sensor 746. FIG. 20clearly shows that the objects 1804, 1806, and 1808 are in differentpositions for the left and right field-of-views 1850 and 1950. Forexample, the object 1804(R) is located to the right of crosshair 1902 aand above crosshair 1902 b while the same object 1804(L) is located tothe left of cross hair 1802 a and further above cross hair 1802 b.

The difference in positions of the objects 1804, 1806, and 1808corresponds to spurious parallax, which is created by deficiencies inthe optical alignment of the optical elements 1402 that produce ZRPs1810 and 1910 in different locations. Assuming no distortion or otherimaging errors, the spurious parallax shown in FIG. 20 is generally thesame for all points within the image. When viewed through oculars of asurgical microscope (such as microscope 200 of FIG. 2), the differencein location of the objects 1804, 1806, and 1808 may not be noticeable.However, when viewed on the display monitors 512 and 514 in astereoscopic image, the differences become readily apparent and canresult in headaches, nausea, and/or vertigo.

FIG. 21 shows a diagram illustrative of spurious parallax with respectto left and right ZRPs. The diagram includes a pixel grid 2100 thatincludes overlays of the right and left pixel grids 1002 and 1004 ofFIG. 10. In this illustrated example, a left ZRP 2102 for the leftoptical path is located at +4 along the x-axis and 0 along the y-axis.In addition, a right ZRP 2104 for the right optical path is located at−1 along the x-axis and 0 along the y-axis. An origin 2106 is shown atthe intersection of the x-axis and the y-axis.

In this example, object 2108 is aligned with respect to the left andright images at a first low magnification. As magnification is increasedby 3×, the object 2108 increased in size and moved away from the ZRPs2102 and 2104. Outlines object 2110 shows a theoretical location of theobject 2108 at the second higher magnification based on the ZRPs 2102and 2104 being aligned with the origin 2106. Specifically, a notch ofthe object 2108 at the first magnification level is at location+2 alongthe x-axis. With 3× magnification, the notch moves 3× along the x-axissuch that the notch is located at +6 along the x-axis at the highermagnification level. In addition, since the ZRPs 2102 and 2104 would betheoretically aligned at the origin 2106, the object 2110 would bealigned between the left and right views (shown in FIG. 21 as a singleobject given the overlay).

However, in this example, misalignment of the left and right ZRPs 2102and 2104 causes the object 2110 to be misaligned between the left andright views at higher magnification. Regarding the right optical path,the right ZRP 2104 is located at −1 along the x-axis such that it is 3pixels away from the notch of the object 2108 at low magnification. Whenmagnified 3×, this difference becomes 9 pixels, which is shown as object2110(R). Similarly, the left ZRP 2102 is located at +4 pixels along thex-axis. At 3× magnification, the object 2108 moves from being 2 pixelsaway to 6 pixels away, which is shown as object 2110(L) at −2 along thex-axis.

The difference in positions of the object 2110(L) and the object 2110(R)corresponds to the spurious parallax between the left and right views atthe higher magnification. If the right and left views were combined intoa stereoscopic image for display, the location of the object 2110 wouldbe misaligned at each row if the renderer program 1850 e uses arow-interleaved mode. The misalignment would be detrimental togenerating stereopsis and may produce an image that appears blurred orconfusing to an operator.

B. Other Sources of Spurious Parallax

While ZRP misalignment between left and right optical paths is asignificant source of spurious parallax, other sources of error alsoexist. For example, spurious parallax may result from non-equalmagnification changes between the right and left optical paths.Differences in magnification between parallel optical paths may resultfrom slight variances in the optical properties or characteristics ofthe lenses of the optical elements 1402. Further, slight differences mayresult from positioning if each of the left and right front zoom lenses726 and 728 and each of the left and right rear zoom lenses 736 and 738of FIGS. 7 and 8 are independently controlled.

Referring back to FIGS. 18 and 19, differences in magnification changeproduce differently sized objects and different spacing between theobjects for the left and right optical paths. If, for example, the leftoptical path has a higher magnification change, then the objects 1804,1806, and 1808 will appear larger and move a greater distance from theZRP 1810 compared to the objects 1804, 1806, and 1808 in the rightfield-of-view 1950 in FIG. 19. The difference in the location of theobjects 1804, 1806, and 1808, even if the ZRPs 1810 and 1910 arealigned, results in spurious parallax.

Another source of spurious parallax results from unequal focusing of theleft and right optical paths. Generally, any difference in focus betweenleft and right views may cause a perceived diminishment in image qualityand potential confusion over whether the left or right view shouldpredominate. If the focus difference is noticeable, it can result in anOut-Of-Focus (“OOF”) condition. OOF conditions are especially noticeablein stereoscopic images where left and right views are shown in the sameimage. In addition, OOF conditions are not easily correctable sincere-focusing an out-of-focus optical path usually results in the otheroptical path becoming unfocused. Generally, a point needs to bedetermined where both optical paths are in focus, which may includechanging positions of left and right lenses along an optical path and/oradjusting a working distance from the target site 700.

FIG. 22 shows a diagram illustrative of how an OOF condition develops.The diagram relates perceived resolution (e.g., focus) to a lensposition relative to an optimal resolution section 2202. In this examplethe left rear zoom lens 734 is at position L1 while the right rear zoomlens 732 is at position R1. At position L1 and R1, the rear zoom lenses732 and 734 are in a range of optimal resolution 2202 such that the leftand right optical paths have matched focus levels. However, there is adifference in the positions of L1 and R1, corresponding to distance ΔP.At a later time, the working distance 706 is changed such that a pointis out-of-focus. In this example, both rear zoom lenses 732 and 734 movethe same distance to locations L2 and R2 such that distance ΔP does notchange. However, the position change results in a significant change inresolution ΔR such that the left rear zoom lens 734 has a higherresolution (e.g., better focus) that the right rear zoom lens 732. Theresolution ΔR corresponds to the OOF condition, which results inspurious parallax from misalignment of focus between the right and leftoptical paths.

Yet another source of spurious parallax can result from imaging objectsthat are moving at the target site 700. The spurious parallax resultsfrom small synchronization errors between exposures of the right andleft optical image sensors 746 and 748. If the left and right views arenot recorded simultaneously, then the object appears to be displaced ormisaligned between the two views. The combined stereoscopic image showsthe same object at two different locations for the left and right views.

Moreover, another source of spurious parallax involves a moving ZRPpoint during magnification. The examples discussed above in SectionIV(A) assume that the ZRPs of the left and right views do not move inthe x-direction or the y-direction. However, the ZRPs may shift duringmagnification if the zoom lenses 726, 728, 732, and/or 734 do not moveexactly parallel with the optical path or axis (e.g., in thez-direction). As discussed above in reference to FIG. 11, the carrier724 may shift or rotate slightly when a force is applied to theactuation section 1108. This rotation may cause the left and right ZRPsto move slightly when a magnification level is changed.

In an example, during a magnification change, the carrier 730 moves in asingle direction while the carrier 724 moves in the same direction for aportion of the magnification change and in an opposite direction for aremaining portion of the magnification change for focus adjustment. Ifthe axis of motion of the carrier 724 is tilted or rotated slightly withrespect to the optical axis, the ZRP of the left and/or right opticalpaths will shift in one direction for the first portion followed by ashift in a reverse direction for the second portion of the magnificationchange. In addition, since the force is applied unequally, the right andleft front zoom lenses 726 and 728 may experience varying degrees of ZRPshift between the left and right optical paths. Altogether, the changein position of the ZRP results in misaligned optical paths, therebyproducing spurious parallax.

C. Reduction in Spurious Parallax Facilitates Incorporating DigitalGraphics and Images with a Stereoscopic View

As surgical microscopes become more digitalized, designers are addingfeatures that overlay graphics, images, and/or other digital effects tothe live-view image. For example, guidance overlays, fusion ofstereoscopic Magnetic Resonance Imaging (“MRI”) images, and/or externaldata may be combined with images recorded by a camera, or even displayedwithin oculars themselves. Spurious parallax reduces the accuracy of theoverlay with the underlying stereoscopic image. Surgeons generallyrequire, for example, that a tumor visualized via MRI be placed asaccurately as possible, often in three dimensions, within a fused livesurgical stereoscopic view. Otherwise, the preoperative tumor imageprovides little information to the surgeon, thereby detracting from theperformance.

For example, a surgical guide may be aligned with a right view imagewhile misaligned with the left view. The misaligned surgical guidebetween the two views is readily apparent to the operator. In anotherexample, a surgical guide may be aligned separately with left and rightviews in the information processor module 1408 prior to the graphicsprocessing unit 1564 creating the combined stereoscopic image. However,misalignment between the left and right views creates misalignmentbetween the guides, thereby reducing the effectiveness of the guides andcreating confusion and delay during the microsurgical procedure.

U.S. Pat. No. 9,552,660, titled “IMAGING SYSTEM AND METHODS DISPLAYING AFUSED MULTIDIMENSIONAL RECONSTRUCTED IMAGE,” (incorporated herein byreference) discloses how preoperative images and/or graphics arevisually fused with a stereoscopic image. FIGS. 23 and 24 show diagramsthat illustrate how spurious parallax causes digital graphics and/orimages to lose accuracy when fused to a stereoscopic image. FIG. 24shows a front view of a patient's eye 2402 and FIG. 23 shows across-sectional view of the eye along plane A-A of FIG. 24. In FIG. 23,the information processor module 1408 is instructed to determine acaudal distance d from a focus plane 2302 to, for example, an object ofinterest 2304 on a posterior capsule of the eye 2402. The informationprocessor module 1408 operates a program 1560 that specifies, forexample, that the distance d is determined by a triangulationcalculation of image data from the left and right views of the eye 2402.A view 2306 is shown from a perspective of the left optical image sensor748 and a view 2308 is shown from a perspective of the right opticalimage sensor 746. The left and right views 2306 and 2308 are assumed tobe coincident with an anterior center 2310 of the eye 2402. In addition,the left and right views 2306 and 2308 are two-dimensional views of theobject 2304 projected onto a focal plane 2302 as theoretical rightprojection 2312 and theoretical left projection 2314. In this example,processor 1562 determines the distance d to the object of interest 2304by calculating an intersection of an extrapolation of the theoreticalright projection 2312 and an extrapolation of the theoretical leftprojection 2314 using a triangulation routine.

However, in this example spurious parallax exists, which causes anactual left projection 2316 to be located to the left of the theoreticalleft projection 2314 by a distance P, as shown in FIGS. 23 and 24. Theprocessor 1562 uses the actual left projection 2316 and the rightprojection 2312 to determine a distance to an intersection 2320 of anextrapolation of the right projection 2312 and an extrapolation of theactual left projection 2316 using the triangulation routine. Thedistance of the intersection point 2320 is equal to the distance d plusan error distance e. The spurious parallax accordingly results in anerroneous distance calculation using data taken from a stereoscopicimage. As shown in FIGS. 23 and 24, even a small degree of spuriousparallax may create a significant error. In the context of a fusedimage, the erroneous distance may result in an inaccurate placement of atumor three-dimensional visualization for fusion with a stereoscopicimage. The inaccurate placement may delay the surgery, hinder theperformance of the surgeon, or cause the entire visualization system tobe disregarded. Worse yet, a surgeon may rely on the inaccurateplacement of the tumor image and make a mistake during the microsurgeryprocedure.

D. The Example Stereoscopic Visualization Camera Reduces or EliminatesSpurious Parallax

The example stereoscopic visualization camera 300 of FIGS. 3 to 16 isconfigured to reduce or eliminate visual defects, spurious parallax,and/or misaligned optical paths that typically result in spuriousparallax. In some examples, the stereoscopic visualization camera 300reduces or eliminates spurious parallax by aligning ZRPs of the left andright optical paths to the respective centers of pixel sets 1006 and1008 of the right and left optical image sensors 746 and 748.Additionally or alternatively, the stereoscopic visualization camera 300may align the optical paths of the left and right images. It should beappreciated that the stereoscopic visualization camera 300 may performactions to reduce spurious parallax during calibration. Additionally,the stereoscopic visualization camera 300 may reduce detected spuriousparallax in real-time during use.

FIGS. 25 and 26 illustrate a flow diagram showing an example procedure2500 to reduce or eliminate spurious parallax, according to an exampleembodiment of the present disclosure. Although the procedure 2500 isdescribed with reference to the flow diagram illustrated in FIGS. 25 and26, it should be appreciated that many other methods of performing thesteps associated with the procedure 2500 may be used. For example, theorder of many of the blocks may be changed, certain blocks may becombined with other blocks, and many of the blocks described areoptional. Further, the actions described in procedure 2500 may beperformed among multiple devices including, for example the opticalelements 1402, the image capture module 1404, the motor and lightingmodule 1406, and/or the information processor module 1408 of the examplestereoscopic visualization camera 300. For example, the procedure 2500may be performed by one of the programs 1560 of the informationprocessor module 1408.

The example procedure 2500 begins when the stereoscopic visualizationcamera 300 receives an instruction to align right and left optical paths(block 2502). The instructions may be received from the user inputdevice 1410 in response to an operator requesting that the stereoscopicvisualization camera 300 perform a calibration routine. In otherexamples, the instructions may be received from the informationprocessor module 1408 after determining right and left images aremisaligned. The information processor module 1408 may determine imagesare not aligned by executing a program 1560 that overlays right and leftimages and determines differences in pixel values, where greaterdifferences over large areas of pixels are indicative of misalignedimages. In some examples, the program 1560 may compare the pixel data ofthe left and right images without performing an overlay function, where,for example, left pixel data is subtracted from right pixel data todetermine a severity of misalignment.

After receiving instructions to reduce spurious parallax, the examplestereoscopic visualization camera 300 locates a ZRP of one of the leftor right optical path. For illustrative purposes, procedure 2500includes the ZRP of the left optical path being determined first.However, in other embodiments, the procedure 2500 may determine the ZRPof the right optical path first. To determine the left ZRP, thestereoscopic visualization camera 300 moves at least one zoom lens(e.g., the left front zoom lens 728 and/or the left rear zoom lens 734)to a first magnification level along a z-direction of the left opticalpath (block 2504). In instances where the front zoom lenses 726 and 728are connected to the same carrier 724 and the rear zoom lenses 732 and734 are connected to the same carrier 730, the movement of the leftlenses causes the right lenses to also move. However, only movement ofthe left lenses is considered during this section of the procedure 2500.

At the first magnification level, the stereoscopic visualization camera300 causes the left zoom lens to move along the z-direction (block2506). The movement may include, for example, back-and-forth movementaround the first magnification level. For example, if the firstmagnification level is 5×, the movement may be between 4× and 6×. Themovement may also include movement in one direction, such as from 5× to4×. During this movement, the stereoscopic visualization camera 300 mayadjust one or more other lenses to maintain focus of the target site700. At block 2508, during the movement of the left zoom lens, thestereoscopic visualization camera 300 records a stream or a sequence ofimages and/or frames 2509 of the target site 700 using, for example, theleft optical image sensor 748. The images 2509 are recorded using anoversized pixel set 1008 configured to encompass an origin of the pixelgrid 1004 and potential locations of the left ZRP.

The example processor 1562 of the information processor module 1408analyzes the image stream to locate a portion of area that does not movein an x-direction or a y-direction between the images (block 2510). Theportion of the area may include one or a few pixels and corresponds tothe left ZRP. As discussed above, during a magnification change, objectsmove away from the ZRP or move towards the ZRP. Only objects at the ZRPremain constant in position with respect to the field-of-view asmagnification changes. The processor 1562 may calculate deltas betweenthe stream of images for each pixel using pixel data. An area with thesmallest delta across the image stream corresponds to the left ZRP.

The example processor 1562 of the information processor module 1408 nextdetermines coordinates of a portion of the area that does not movebetween the image stream (e.g., determines a location of the left ZRP)with respect to the pixel grid 1004 (block 2512). In other examples, theprocessor 1562 of the information processor module 1408 determines adistance between the origin and the portion of the area corresponding tothe left ZRP. The distance is used to determine a position of the leftZRP on the pixel grid 1004. Once the location of the left ZRP isdetermined, the processor 1562 of the information processor module 1408determines a pixel set (e.g., the pixel set 1008) for the left opticalimage sensor 748 such that the left ZRP is located at a center (withinone pixel) of the pixel set (block 2514). At this point, the left ZRP iscentered within the left optical path.

In some examples, blocks 2504 to 2514 may be performed iteratively byre-selecting the pixel set until the left ZRP is within a pixel of theorigin and spurious parallax is minimized. After the pixel grid isdetermined, the processor 1562 of the information processor module 1408stores at least one of coordinates of the pixel set and/or coordinatesof the left ZRP to the memory 1570 as a calibration point (block 2516).The processor 1562 of the information processor module 1408 mayassociate the first magnification level with the calibration point suchthat the same pixel set is selected when the stereoscopic visualizationcamera 300 returns to the first magnification level.

FIG. 27 shows a diagram illustrative of how the left ZRP is adjustedwith respect to the pixel grid of the left optical image sensor 748.Initially, an initial (e.g., oversized) pixel set 2702 is selected,which is centered on origin 2704. The pixel set 2702 is large enough torecord potential ZRPs in the image stream. In this illustrated example,a left ZRP 2706 is located above and to the right of the origin 2704.The processor 1562 of the information processor module 1408 determinespixel set 2708 based on a location of the left ZRP 2706 such that theleft ZRP 2706 is located or positioned at a center of the pixel set2708.

After the left ZRP is determined and aligned with an origin of a pixelset in FIG. 25, the example procedure 2500 aligns the left and rightimages in FIG. 26. To align the images, the example processor 1562compares pixel data from left and right images recorded after the leftZRP is aligned with the origin. In some embodiments, the processor 1562overlays the left and right images to determine differences using, forexample, a subtraction and/or template method. The processor 1562selects or determines a pixel set for the right optical path such thatthe resulting right images align or coincide with the left images (block2519).

The example processor 1562, in the illustrated embodiment, determinesthe right ZRP. The steps are similar to steps discussed in blocks 2504to 2512 for the left ZRP. For example, at block 2518 the stereoscopicvisualization camera 300 moves a right zoom lens to the firstmagnification level. In some embodiments, the magnification level forthe right lens is different than the magnification level used fordetermining the left ZRP. The example processor 1562 of the informationprocessor module 1408 then moves the right zoom lens around themagnification level and receives a stream of images 2521 from the rightoptical image sensor 746 during the movement (blocks 2520 and 2522). Theexample processor 1562 of the information processor module 1408determines the right ZRP from the right stream of images by locating aportion of an area that does not move between the images (block 2524).The processor 1562 next determines coordinates of the right ZRP and/or adistance between a center of an aligned pixel set 1006 to the right ZRP(block 2526).

The processor 1562 then instructs the motor and lighting module 1406 tomove at least one lens in the right optical path in at least one of anx-direction, a y-direction, and/or a tilt-direction to align the rightZRP with the center of the aligned pixel set 1006 using, for example,the distance or coordinates of the right ZRP (block 2528). In otherwords, the right ZRP is moved to coincide with the center of the alignedpixel set 1006. In some examples, the right front lens 720, the rightlens barrel 736, the right final optical element 745, and/or the rightimage sensor 746 is moved (using for example a flexure) in thex-direction, the y-direction and/or a tilt-direction with respect to thez-direction of the right optical path. The degree of movement isproportional to the distance of the right ZRP from the center of thepixel set 1006. In some embodiments, the processor 1562 digitallychanges properties of the right front lens 720, the right lens barrel736, and/or the right final optical element 745 to have the same effectas moving the lenses. The processor 1562 may repeat steps 2520 to 2528and/or use subsequent right images to confirm the right ZRP is alignedwith the center of the pixel set 1006 and/or to iteratively determinefurther lens movements needed to align the right ZRP with the center ofthe pixel set.

The example processor 1562 stores coordinates of the right pixel setand/or the right ZRP to the memory 1570 as a calibration point (block2530). The processor 1562 may also store to the calibration point aposition of the right lens that was moved to align the right ZRP. Insome examples, the calibration point for the right optical path isstored with the calibration point for the left optical path inconjunction with the first magnification level. Thus, the processor 1562applies the data within the calibration point to the optical imagesensors 746 and 748 and/or radial positioning of one or more opticalelements 1402 when the stereoscopic visualization camera 300 issubsequently set to the first magnification level.

In some examples, the procedure 2500 may be repeated for differentmagnification levels and/or working distances. Accordingly, theprocessor 1562 determines if ZRP calibration is needed for anothermagnification level or working distance (block 2532). If anothermagnification level is to be selected, the procedure 2500 returns toblock 2504 in FIG. 25. However, if another magnification level is notneeded, the example procedure ends.

Each of the calibration points may be stored in a look-up-table. Eachrow in the table may correspond to a different magnification leveland/or working distance. Columns in the look-up-table may providecoordinates for the left ZRP, the right ZRP, the left pixel set, and/orthe right pixel set. In addition, one or more columns may specifyrelevant positions (e.g., radial, rotational, tilt, and/or axialpositions) of the lenses of the optical elements 1402 to achieve focusat the magnification level in addition to aligned right and left images.

The procedure 2500 accordingly results in the right ZRP and the left ZRPin addition to views of the target site to be aligned to pixel grids ofthe respective optical image sensors 746 and 748 as well as to eachother in a three-dimensional stereoscopic image. In some instances, theleft and right images and the corresponding ZRPs have an accuracy andalignment to within one pixel. Such accuracy may be observable on thedisplay 514 or 514 by overlaying left and right views (e.g., images fromthe left and right optical paths) and observing both views with botheyes, rather than stereoscopically.

It should be appreciated that in some examples, a right pixel set isfirst selected such that the right ZRP is aligned with or coincidentwith an origin of the pixel set. Then, the right and left optical imagesmay be aligned by moving one or more right and/or left lenses of theoptical elements 1402. This alternative procedure still provides rightand left ZRPs that are centered and aligned between each other and withrespect to the optical image sensors 746 and 748.

The procedure 2500 ultimately reduces or eliminates spurious parallax inthe stereoscopic visualization camera 300 throughout a full opticalmagnification range by ensuring left and right ZRPs remain aligned andthe right and left images remain aligned. In other words, the dualoptics of the right and left optical images sensors 746 and 748 arealigned such that parallax at a center of an image between the left andright optical paths is approximately zero at the focal plane.Additionally, the example stereoscopic visualization camera 300 is parfocal across the magnification range, and par central acrossmagnification and working distance ranges since the ZRP of each opticalpath has been aligned to a center of the respective pixel set.Accordingly, changing only the magnification will maintain a focus ofthe target site 700 in both optical image sensors 746 and 748 whilebeing trained on the same center point.

The above procedure 2500 may be performed at calibration before asurgical procedure is performed and/or upon request by an operator. Theexample procedure 2500 may also be performed prior to image registrationwith a pre-operative microsurgical image and/or surgical guidancegraphics. Further, the example procedure 2500 may be performed inreal-time automatically during operation of the stereoscopicvisualization camera 300.

1. Template Matching Example

In some embodiments, the example processor 1562 of the informationprocessor module 1408 is configured to use a program 1560 in conjunctionwith one or more templates to determine a position of the right ZRPand/or the left ZRP. FIG. 28 shows a diagram illustrative of how theprocessor 1562 uses a target template 2802 to determine a location of aleft ZRP. In this example, FIG. 28 shows a first left image includingthe template 2802 aligned with an origin 2804 or center of the leftpixel grid 1004 of the left optical image sensor 748. The template 2802may be aligned by moving the stereoscopic visualization camera 300 tothe appropriate location. Alternatively, the template 2802 may be movedat the target site 700 until aligned. In other examples, the template2802 may include another pattern that does not need alignment with acenter of the pixel grid 1004. For example, the template may include agraphical wave pattern, a graphical spirograph pattern, a view of asurgical site of a patient and/or a grid having visually distinguishablefeatures with some degree of non-periodicity in both the x andy-directions. The template is configured to prevent a subset of aperiodic image from being perfectly aligned onto the larger image in aplurality of locations, which makes such templates unsuitable formatching. A template image that is suitable for template matching isknown as a “template match-able” template image.

The template 2802 shown in FIG. 28 is imaged at a first magnificationlevel. A left ZRP 2806 is shown with respect to the template 2802. TheZRP 2806 has coordinates of L_(x), L_(y) with respect to the origin2804. However, at this point in time, the processor 1562 has not yetidentified the left ZRP 2806.

To locate the ZRP 2806, the processor 1562 causes a left zoom lens(e.g., the left front zoom lens 728 and/or the left rear zoom lens 734)to change magnification from the first magnification level to a secondmagnification level, specifically in this example, from 1× to 2×. FIG.29 shows a diagram of a second left image including the target 2802 onthe pixel grid 1004 with the magnification level doubled. From the firstmagnification level to the second magnification level, portions of thetarget 2802 increase in size and expand uniformly away from the left ZRP2806, which remains stationary with respect to the first and secondimages. In addition, a distance between the origin 2804 of the pixelgrid 1004 and the left ZRP 2806 remains the same.

The example processor 1562 synthesizes a digital template image 3000from the second image shown in FIG. 29. To create the digital templateimage, the processor 1562 copies the second image shown in FIG. 29 andscales the copied image by the reciprocal of the magnification changefrom the first to the second magnification. For example, if themagnification change from the first image to the second image was by afactor of 2, then the second image is scaled by ½. FIG. 30 shows adiagram of the digital template image 3000, which includes the template2802. The template 2802 in the digital template image 3000 of FIG. 30 isscaled to be the same size as the template 2802 in the first left imageshown in FIG. 28.

The example processor 1562 uses the digital template image 3000 tolocate the left ZRP 2806. FIG. 31 shows a diagram that shows the digitaltemplate image 3000 superimposed on top of the first left image (or asubsequent left image recorded at the first magnification level)recorded in the pixel grid 1004. The combination of the digital templateimage 3000 with the first left image produces a resultant view, asillustrated in FIG. 31. Initially the digital template image 3000 iscentered at the origin 2804 of the pixel grid 1004.

The example processor 1562 compares the digital template image 3000 tothe underlying template 2802 to determine if they are aligned ormatched. The example processor 1562 then moves the digital templateimage 3000 one or more pixels either horizontally or vertically andperforms another comparison. The processor 1562 iteratively moves thedigital template image 3000 compiling a matrix of metrics for eachlocation regarding how close the digital template image 3000 matches theunderlying template 2802. The processor 1562 selects the location in thematrix corresponding to the best matching metric. In some examples, theprocessor 1562 uses the OpenCV™ Template Match function.

FIG. 32 shows a diagram with the digital template image 3000 alignedwith the template 2802. The distance that the digital template image3000 was moved to achieve optimal matching is shown as Δx and Δy.Knowing the digital template image 3000 was synthesized at a scale ofM1/M2 (the first magnification level divided by the second magnificationlevel), the processor 1562 determines the coordinates (L_(x), L_(y)) ofthe left ZRP 2806 using Equations (1) and (2) below.

Lx=Δx/(M1/M2)  Equation (1)

Ly=Δy/(M1/M2)  Equation (2)

After the coordinates (Lx, Ly) of the left ZRP 2806 are determined, theexample processor 1562 selects or determines a pixel subset with anorigin that is aligned or coincides with the left ZRP 2806, as discussedabove in conjunction with procedure 2500 of FIGS. 25 and 26. In someembodiments, the processor 1562 may use template matching iteratively toconverge on a highly accurate ZRP position and/or pixel subset. Further,while the above example discussed locating the left ZRP, the sametemplate matching procedure can be used to locate the right ZRP.

In some embodiments, the above-described template matching program 1560may be used to align the left and right images. In these embodiments,left and right images are recorded at a magnification level. Both theimages may include, for example, the target template 2802 of FIG. 28. Aportion of the right image is selected and overlaid with the left image.The portion of the right image is then shifted around the left image byone or more pixels horizontally and/or vertically. The example processor1562 performs a comparison at each location of the portion of the rightimage to determine how close a match exists with the left image. Once anoptimal location is determined, a pixel set 1006 of the right pixel grid1002 is determined such that the right image is generally coincidentwith the left image. The location of the pixel set 1006 may bedetermined based on how much the portion of the right image was moved tocoincide with the left image. Specifically, the processor 1562 uses anamount of movement in the x-direction, the y-direction, and/or thetilt-direction to determine corresponding coordinates for the rightpixel set 1006.

2. Right and Left Image Alignment Example

In some embodiments, the example processor 1562 of the informationprocessor module 1408 of FIGS. 14 to 16 displays an overlay of right andleft images on the display monitor 512 and/or 514. The processor 1562 isconfigured to receive user feedback for aligning the right and leftimages. In this example each pixel data for the right and left images isprecisely mapped to a respective pixel of the display monitor 512 using,for example, the graphics processing unit 1564. The display of overlaidleft and right images makes any spurious parallax readily apparent to anoperator. Generally, with no spurious parallax, the left and rightimages should almost exactly align.

If an operator detects spurious parallax, the operator may actuatecontrols 305 or the user input device 1410 to move either the right orleft image for alignment with the other of the right and left image.Instructions from the controls 305 may cause the processor 1562 toaccordingly adjust the location of the left or right pixel set inreal-time, such that subsequent images are displayed on the displaymonitor 512 reflective of the operator input. In other examples, theinstructions may cause the processor 1562 to change a position of one ormore of the optical elements 1402 via radial adjustment, rotationaladjustment, axial adjustment, or tilting. The operator continues toprovide input via controls 305 and/or the user input device 1410 untilthe left and right images are aligned. Upon receiving a confirmationinstruction, the processor 1562 stores a calibration point to alook-up-table reflective of the image alignment at the set magnificationlevel.

Additionally or alternatively, the template match method described abovemay be used to perform image alignment while focused on a planar targetthat is approximately orthogonal to a stereo optical axis of thestereoscopic visualization camera 300. Moreover, the template matchmethod may be used to align the left and right views in real-timewhenever a “template match-able” scene is in view of both the left andright optical paths. In an example, a template image is copied from asubset of, for instance, the left view, centered upon or near the centerof the view. Sampling from the center for an in-focus image ensures thata similar view of the target site 700 will be present in the other view(in this example the right view). For out-of-focus images, this is notthe case such that in the current embodiment this alignment method isperformed only after a successful auto-focus operation. The selectedtemplate is then matched in the current view (or a copy thereof) of theother view (in this example the right view) and only a y-value is takenfrom the result. When the views are aligned vertically, the y-value ofthe template match is at or near zero pixels. A non-zero y-valueindicates vertical misalignment between the two views and a correctionusing the same value of y is applied either to select the pixel readoutset of the first view or a correction using the negated value of y isapplied to the pixel readout set of the other view. Alternatively, thecorrection can be applied in other portions of the visualizationpipeline, or split between pixel readout set(s) and said pipeline.

In some examples, the operator may also manually align a right ZRP withan origin of the pixel grid 1002. For instance, after determining alocation of the right ZRP, the processor 1562 (and/or the peripheralinput unit interface 1574 or graphics processing unit 1564) causes theright ZRP to be highlighted graphically on a right image displayed bythe display monitor 512. The processor 1562 may also display a graphicindicative of the origin of the pixel grid 1002. The operator usescontrols 305 and/or the user input device 1410 to steer the right ZRP tothe origin. The processor 1562 uses instructions from the controls 305and/or the user input device 1410 to accordingly move one or more of theoptical elements 1402. The processor 1562 may provide a stream of rightimages in real-time in addition to graphically displaying the currentlocation of the right ZRP and origin to provide the operator updatedfeedback regarding positioning. The operator continues to provide inputvia controls 305 and/or the user input device 1410 until the right ZRPis aligned. Upon receiving a confirmation instruction, the processor1562 stores a calibration point to a look-up-table reflective ofpositions of the optical elements 1402 at the set magnification level.

3. Comparison of Alignment Error

The example stereoscopic visualization camera 300 produces lessalignment error between right and left images compared to known digitalsurgical microscopes with stereoscopic cameras. The analysis discussedbelow compares spurious parallax generated by ZRP misalignment for aknown digital surgical microscope with camera and the examplestereoscopic visualization camera 300. Initially, both cameras are setat a first magnification level with a focal plane positioned on a firstposition of a patient's eye. Equation (3) below is used to determineworking distance (“WD”) from each camera to the eye.

WD=(IPD/2)/tan(α)  Equation (3)

In this equation, IPD corresponds to the interpupillary distance, whichis approximately 23 mm. In addition, a is one-half of an angle between,for example, the right optical image sensor 746 and the left opticalimage sensor 748, which is 2.50° in this example. The convergence angleis two times this angle, which is 5°, in this example. The resultingworking distance is 263.39 mm.

The cameras are zoomed in to a second magnification level andtriangulated on a second position of the patient's eye. In this examplethe second position is at the same physical distance from the camera asthe first position, but presented at the second magnification level. Thechange in magnification generates spurious horizontal parallax due tomisalignment of one or both of the ZRPs with respect to a center of asensor pixel grid. For the known camera system, the spurious parallax isdetermined to be, for example, 3 arc-minutes, which corresponds to0.05°. In Equation (3) above, the 0.05° value is added to a, whichproduces a working distance of 258.22 mm. The difference in workingdistance is 5.17 mm (263.39 mm−258.22 mm), which corresponds to theerror of the known digital surgical microscope with camera attachment.

In contrast, the example stereoscopic visualization camera 300 iscapable of automatically aligning ZRPs to be within one pixel of acenter of a pixel set or grid. If the angular field-of-view is 5° andrecorded with a 4k image sensor used in conjunction with a 4k displaymonitor, the one pixel accuracy corresponds to 0.00125° (5°/4000) or 4.5arc-seconds. Using Equation (3) above, the 0.00125° value is added to a,which produces a working distance of 263.25 mm. The difference inworking distance for the stereoscopic visualization camera 300 is 0.14mm (263.39 mm−263.25 mm). When compared to the 5.17 mm error of theknown digital surgical microscope, the example stereoscopicvisualization camera 300 reduces alignment error by 97.5%.

In some embodiments, the stereoscopic visualization camera 300 may bemore accurate at higher resolutions. In the example above, theresolution is about 4.5 arc-seconds for a 5° field-of-view. For an 8Kultra-high definition system (with 8000 pixels in each of 4000 rows)with a field-of-view of 2°, the resolution of the stereoscopicvisualization camera 300 is approximately 1 arc-second. This means thatZRP of the left and right views may be aligned to one pixel or 1arc-second. This is significantly more precise than known digitalmicroscope systems that have spurious parallax on the order ofarc-minutes.

4. Reduction of Other Sources of Spurious Parallax

The above-examples discuss how the example stereoscopic visualizationcamera 300 reduces spurious parallax as a result of misaligned ZRPsand/or left and right images themselves. The stereoscopic visualizationcamera 300 may also be configured to reduce other sources of spuriousparallax. For example, the stereoscopic visualization camera 300 mayreduce spurious parallax due to motion by simultaneously clocking theright and left optical image sensors 746 and 748 to record images atsubstantially the same instant.

The example stereoscopic visualization camera 300 may also reducespurious parallax due to dissimilar magnification between the left andright optical paths. For example, the stereoscopic visualization camera300 may set the magnification level based on the left optical path. Thestereoscopic visualization camera 300 may then make automaticadjustments so that the magnification of the right image matches theleft. The processor 1562, for example, may use image data to calculatecontrol parameters, for example by measuring a number of pixels betweencertain features common in the left and right images. The processor 1562may then equalize the magnification levels of the left and right imagesby digital scaling, inserting interpolative pixels, and/or deletingextraneous pixels. The example processor 1562 and/or the graphicsprocessing unit 1564 may re-render the right image such that themagnification is matched to the left image. Additionally oralternatively, the stereoscopic visualization camera 300 may includeindependent adjustment of the left and right optical elements 1402. Theprocessor 1562 may separately control the left and right opticalelements 1402 to achieve the same magnification. In some examples, theprocessor 1562 may first set, for example, the left magnification levelthen separately adjust the right optical elements 1402 to achieve thesame magnification level.

The example stereoscopic visualization camera 300 may further reducespurious parallax due to dissimilar focus. In an example, the processor1562 may execute a program 1560 that determines a best focus for eachoptical path for a given magnification and/or working distance. Theprocessor 1562 first performs a focusing of the optical elements 1402 ata point of best resolution. The processor 1562 may then check the OOFcondition at a suitable non-object-plane location and match the focusfor the left and right images. The processor 1562 next re-checks thefocus at best resolution and adjusts the focus iteratively until bothleft and right optical elements 1402 focus equally well both on and awayfrom an object plane.

The example processor 1562 may measure and verify optimal focus bymonitoring a signal relating to the focus of one or both of the rightand left images. For example, a “sharpness” signal is generated by thegraphics processing unit 1564 for the left and right imagessimultaneously and/or in synchronization. The signal changes as focuschanges and may be determined from an image-analysis program, an edgedetection analysis program, a bandwidth of Fourier transforms of patternintensity program, and/or a modulation transfer function (“MTF”)measurement program. The processor 1562 adjusts a focus of the opticalelements 1402 while monitoring for a maximum signal indicative of asharp image.

To optimize the OOF condition, the processor 1562 may monitor sharpnesssignals for both the left and right images. If the focus is moved off ofthe object plane and the signal related to, for example, the left imageincreases but the signal related to the right image decreases, theprocessor 1562 is configured to determine the optical elements 1402 aremoving out of focus. However, if the signals related to both the rightand left images are relatively high and approximately equal, theprocessor 1562 is configured to determine the optical elements 1402 areproperly positioned for focusing.

5. Benefits of Low Spurious Parallax

The example stereoscopic visualization camera 300 has a number ofadvantages over known digital surgical microscopes as a result of thelow spurious parallax between right and left images. For example, almostperfectly aligned left and right images produce an almost perfectstereoscopic display for a surgeon, thereby reducing eye fatigue. Thisallows the stereoscopic visualization camera 300 to be used as anextension of a surgeon's eyes rather than a cumbersome tool.

In another example, precisely aligned left and right images allowaccurate measurements of the surgical site to be digitally taken. Forinstance, a size of a patient's ocular lens capsule may be measured suchthat a properly-sized IOL can be determined and accurately implanted. Inanother instance, a motion of a moving blood vessel may be measured suchthat an infrared fluorescein overlay can be accurately placed in a fusedimage. Here, the actual motion velocity is generally not of interest tothe surgeon but critical for the placement and real-time adjustment ofthe overlaid image. Properly matched scale, registration, andperspective of the overlaid images are all important to provide anaccurately-fused combined live stereoscopic image and an alternate-modeimage.

In some examples, the processor 1562 may enable an operator to drawmeasurement parameters on the display monitor 512. The processor 1562receives the drawn coordinates on a screen and accordingly translatesthe coordinates to the stereoscopic image. The processor 1562 maydetermine measurement values by scaling the drawn ruler on the displaymonitor 512 to a magnification level shown in the stereoscopic images.The measurements made by the processor 1562 include point-to-pointmeasurements of two or three locations displayed in the stereoscopicdisplay, point-to-surface measurements, surface characterizationmeasurements, volume determination measurements, velocity verificationmeasurements, coordinate transformations, instrument and/or tissuetracking, etc.

VII. Example Robotics System for the Stereoscopic Visualization Camera

As discussed in connection with FIGS. 5 and 6, an example stereoscopicvisualization camera 300 may be connected to a mechanical or robotic arm506 as part of a stereoscopic visualization platform or stereoscopicrobotic platform 516. The example robotic arm 506 is configured toenable an operator to position and/or orient the stereoscopicvisualization camera 300 above and/or next to a patient during one ormore procedures. Accordingly, the robotic arm 506 enables an operator tomove the stereoscopic visualization camera 300 to a desiredfield-of-view (“FOV”) of a target surgical site. Surgeons generallyprefer cameras to be positioned and/or orientated in a FOV that issimilar to their own FOV to enable easier visual orientation andcorrespondence between images displayed on a screen and the surgeon'sFOV. The example robotic arm 506 disclosed herein provides structuralflexibility and assisted control to enable positioning to coincide or tobe consistent with a surgeon's FOV without blocking the surgeon's ownFOV.

In contrast to the stereoscopic robotic platform 516 disclosed herein,known stereomicroscope holding devices include a simple mechanical armthat is manually moved by an operator. These devices include multiplerotational joints equipped with electromechanical brakes that allowmanual repositioning. Further, to allow an operator to change a vieweasily and without interrupting a procedure, some known holding deviceshave motorized joints. The motorized joints have various levels ofcomplexity ranging from, for example, simple X-Y positioning up todevices comprising multiple independent rotational joints thatmanipulate connected rigid arms. During most procedures, it is desirableto obtain views from various directions quickly and easily. However,known stereomicroscope holding devices suffer from one or more problems.

Known stereomicroscope holding devices have limited position, direction,and/or orientation accuracy that is generally limited by the manualability of the surgeon to manipulate the microscope to view desirableaspects of the image. Holding devices with multiple joints can beespecially cumbersome to operate since device manipulation usuallyresults in all the joints moving at the same time. Oftentimes, anoperator is watching how an arm moves. After the arm is positioned in adesired location, the operator checks whether the imaging device's FOVis aligned in the desired location. Many times, even if the device isaligned properly, a focus of the device has to be adjusted. Furtherknown stereomicroscope holding devices cannot provide consistent FOV orfocal planes with respect to other objects in a targeted surgical sitebecause the devices do not have arm position memories, or the memoriesare inaccurate as a patient is moved or shifted during a procedure.

Known stereomicroscope holding devices generally have positioningsystems in which control is independent of microscope parameters such asobject plane focus distance, magnification and illumination. For thesedevices, coordination of positioning and, for example, zooming must beperformed manually. In an example, an operator may reach a lens limitfor focusing or changing a working distance. The operator has tomanually change a position of the holding device, and then refocus thestereomicroscope.

Known stereomicroscope holding devices are intended solely forobservation of a surgical site. The known devices do not determinelocations or distances from tissue within a FOV to another objectoutside the FOV. The known devices also do not provide comparisons oftissue with other objects within a live surgical site to form analternative viewing modality, such as combining an MM image with a liveview. Instead, views from known devices are displayed separately andunaligned from other medical images or templates.

Additionally, known stereomicroscope holding devices have parametersthat may not be accurate since there is less emphasis on precision,other than for observation. The requirements in ISO Standard10936-1:2000(E) “Optics and optical instruments—Operationmicroscopes—Part 1: Requirements and test methods”, have largely beenderived to achieve reasonable stereoscopic optical image quality usingoculars by a normal human operator. The operator's brain combines theviews into the mind's image to achieve stereopsis. The views aregenerally not combined or compared together otherwise. As long as theoperator sees an acceptable image and does not suffer from deleteriouseffects like headaches their needs have been met. The same is true forstereomicroscope holding devices, where some instability, arm sag, andimprecise movement control are permitted. However, when high-resolutiondigital cameras are used with known holding devices, the structuralinaccuracies are readily observable and may detract from their use,especially for microsurgical procedures.

As mentioned above, known stereomicroscope holding devices may sag dueto the weight of a camera. Generally, known robotic positioning systemsare calibrated to determine compliance or inaccuracy only for the systemitself. The stereomicroscope holding devices do not take into accountthe camera or any inaccuracy between a camera mount and the holdingdevice. Sag is generally compensated by an operator manually positioningthe camera while observing an image on a display. In systems thatprovide motorized motion, changes in sag occur, for example, when acenter-of-gravity (“CG”) of the camera is re-positioned on an oppositeside of a rotational axis of an arm joint, where restoring torque momentabout the axis reverses direction. Subsequently any compliance or sag inthe mechanism, which is compensated by an operator by adjusting theposition, direction, and/or orientation of the camera, now adds to theposition, direction, and/or orientation error. In some cases, forexample when the camera is moved through a robotic singularity point,the moment reversal occurs quickly and the resulting camera image shiftsin error quickly and excessively. Such error limits the ability of knownstereomicroscope holding devices to, for example, accurately follow ortrack tissue or instruments in the site.

Known stereomicroscope holding devices include features for spatiallylocating and tracking surgical instruments and providing theirsubsequent representative display on a monitor. However, these knownsystems require an additional stereoscopic locating camera ortriangulation device, prominently located, as well as conspicuousfiducial devices on the instruments. The added devices add tocomplexity, cost and operational obtrusiveness.

The example stereoscopic robotic platform 516 disclosed herein includesan example stereoscopic visualization camera 300 connected to amechanical or robotic arm 506. FIGS. 5 and 6 illustrate an example ofthe stereoscopic robotic platform 516. Stereoscopic images recorded bythe camera 300 are displayed via one or more display monitors 512, 514.The robotic arm 506 is mechanically connected to a cart 510, which mayalso support one or more of the display monitors 512, 514. The roboticarm may include, for example, an articulated robotic arm generallyanthropomorphic in size, nature, function, and operation.

FIG. 33 shows a side-view of the microsurgical environment 500 of FIG.5, according to an example embodiment of the present disclosure. In theillustrated example, the display monitor 512 may be connected to thecart 510 via a mechanical arm 3302 with one or more joints to enableflexible positioning. In some embodiments, the mechanical arm 3302 maybe long enough to extend over a patient during surgery to providerelatively close viewing of a surgeon.

FIG. 33 also illustrates a side-view of the stereoscopic roboticplatform 516, including the stereoscopic visualization camera 300 andthe robotic arm 506. The camera 300 is mechanically coupled to therobotic arm 506 via a coupling plate 3304. In some embodiments, thecoupling plate 3304 may include one or more joints that provide forfurther degrees of positioning and or orientation of the camera 300. Insome embodiments, the coupling plate 3304 has to be manually moved orrotated by an operator. For example, the coupling plate 3304 may have ajoint that enables the camera 300 to be positioned quickly betweenhaving an optical axis along a z-axis (i.e., pointing downward toward apatient) and an optical axis along an x-axis or y-axis (i.e., pointingsideward toward a patient).

The example coupling plate 3304 may include a sensor 3306 configured todetect forces and/or torques imparted by an operator for moving thecamera 300. In some embodiments, an operator may position the camera 300by gripping the control arms 304 a and 304 b (shown in FIG. 3). Afterthe operator has clutched the control arms 304 a and 304 b with theirhands, the user may position and/or orient the camera 300 withassistance from the robotic arm 306. The sensor 3306 detects a forcevector or torque angle provided by the operator. The example platform516 disclosed herein uses the sensed force/torque to determine whichjoints of the robotic arm 506 should be rotated (and how quickly thejoints should be rotated) to provide assisted movement of the camera 300that corresponds to the forces/torques provided by the operator. Thesensor 3306 may be located at an interface between the coupling plate3304 and the camera 300 for detecting the forces and/or torques impartedby an operator via the control arms 304.

In some embodiments, the sensor 3306 may include, for example, asix-degrees-of-freedom haptic force-sensing module. In theseembodiments, the sensor 3306 may detect translational force or motion inthe x-axis, y-axis, and z-axis. The sensor 3306 may also separatelydetect rotational force or motion around a yaw-axis, a pitch-axis, and aroll-axis. The decoupling of the translational force and the rotationalforce may enable the stereoscopic robotic platform 516 to more easilycalculate direct and/or reverse kinematics for control of the robot arm506.

The example sensor 3306 may be configured to detect force since therobotic arm 506 may not be movable by a user alone. Instead, the sensor3306 detects translational and rotational force applied by a user, whichis used by the stereoscopic robotic platform 516 to determine whichjoints to rotate to provide assisted movement control of the robotic arm506. In other examples, the robotic arm 506 may permit operator movementwithout assistance, or at least initial assistance. In these otherexamples, the sensor 3306 detects motion imparted by the user, which isused by the stereoscopic robotic platform 516 to subsequently cause oneor more joints to rotate, thereby providing assisted movement. The timebetween initial detection of motion or the force resulting in themotion, until the stereoscopic robotic platform 516 causes the joints torotate may be less than 200 milliseconds (“ms”), 100 ms, 50 ms, or asfew as 10 ms, where the user does not notice the initial time ofunassisted movement of the robotic arm 506.

The example sensor 3306 may output digital data that is indicative ofthe rotational force/motion and digital data that is indicative of thetranslational force/motion. In this example, the digital data may have8, 16, 32, or 64 bit resolution for the detected force/motion in eachaxis. Alternatively, the sensor 3306 may transmit an analog signal thatis proportional to the sensed force and/or motion. The example sensor3306 may transmit the data at a periodic sampling rate of, for example,1 ms, 5 ms, 10 ms, 20 ms, 50 ms, 100, ms, etc. Alternatively, the sensor3306 may provide a near-continuous stream of force/motion data.

In some embodiments, the example sensor 3306 may instead be located inone or more of the control arms 304 a and 304 b or between the controlarms 304 a and 304 b and the housing 302. In examples, where each of thecontrol arms 304 a and 304 b include sensor 3306, the examplestereoscopic robotic platform 516 may receive two sets of translationaland rotational force or motion. In these examples, the stereoscopicrobotic platform 516 may average the values from the sensors 3306.

In the illustrated embodiments, a first end of the robotic arm 506 ismounted to the cart 510 while a second, opposite end of the robotic armis mechanically connected to stereoscopic visualization camera 300(e.g., a robot end effector). FIG. 33 shows the robotic arm 506 holdingthe stereoscopic visualization camera 300 in an extended position, suchas positioning the stereoscopic visualization camera 300 above asurgical site while keeping the rest of the platform 516 out of the wayof a surgeon. The cart 510 is configured to securely hold thestereoscopic robotic platform 516 and is weighted and balanced toprevent tipping under prescribed operating positions.

The example stereoscopic robotic platform 516 is configured to providethe following benefits.

-   -   1. Enhanced visualization. Communication between the robotic arm        506 and the stereoscopic visualization camera 300 enables the        platform 516 to point and steer the camera 300 to quickly and        more accurately visualize surgical sites. For example, the        robotic arm 506 can move the camera 300 along its optical axis        to extend the range of focusing and zooming beyond that        contained in just the camera. The relatively small size of the        platform 516 provides for Heads-Up Surgery® in a wider variety        of surgical procedures and orientations, thereby improving        surgical efficiency and surgeon ergonomics.    -   2. Enhanced dimensional performance. The example stereoscopic        visualization camera 300, with its accurate measurement        capability of all points within the stereoscopic image, is        configured to communicate the measurement information to the        robotic arm 506. The robotic arm 506, in turn, comprises        accurate position, direction, and/or orientation determination        capability and is registered to the camera 300 such that the        dimensions within and between images can be accurately        transformed respective to a coordinate system common to the        stereoscopic robotic platform 516 and an anatomy of a patient.    -   3. The quality and accuracy of stereoscopic image data from the        stereoscopic visualization camera 300 enables it to be combined        with image or diagnostic data from external sources of various        modalities to construct fused images. Such fused images can be        used by surgeons to perform procedures more accurately and        efficiently and to achieve better patient outcomes.    -   4. The stereoscopic visualization camera 300, the robotic arm        506, and/or image and motion processors (e.g., processor 1408 of        FIG. 14) can be programmed for beneficial procedural        applications. For example, a specific visualization site        position, direction, and/or orientation can be saved, and then        returned to later in the procedure. Precise motion paths can be        programmed to, for example, follow a specific length or line of        tissue. In other examples, pre-programmed waypoints can be set,        thereby permitting an operator to change a position and/or        orientation of the robotic arm 506 based upon which step is        being performed during a medical procedure.    -   5. The stereoscopic robotic platform 516 provides for        intrinsically guided surgery by use and analysis of accurate        image position information. Such guidance can be communicated to        other devices, such as another robotic system that performs at        least portions of a surgical procedure. Components of the        stereoscopic robotic platform 516 that share functionality with        components of such other devices may be integrated together into        a package to achieve efficiencies of performance, accuracy, and        cost.

A. Robotic Arm Embodiment

FIG. 34 illustrates an embodiment of the example robotic arm 506,according to an example embodiment of the present disclosure. In someembodiments, the robotic arm 506 is similar to or comprises model UR5from Universal Robots S/A. The exterior surfaces of the robotic arm 506comprise aluminum and plastic materials, which are compatible for use inan operating room and easily cleaned.

While the robotic arm 506 is described herein as beingelectromechanical, in other examples, the robotic arm 506 may bemechanical, hydraulic, or pneumatic. In some embodiments, the roboticarm 506 may have mixed actuation mechanisms, for example, using a vacuumchuck with a control valve to hold and manipulate the camera 300.Further, while the robotic arm 506 is described below as including acertain number of joints and links, it should be appreciated that therobotic arm 506 may include any number of joints, any lengths of links,and/or comprise any types of joints, or sensors.

As described herein, the robotic arm 506 is situated and the joints areoriented to provide an unrestricted view of an operating field whileproviding a 3D stereoscopic display for an operator for any surgicalprocedure for a patient. Movement of the robotic arm 506 in noncriticalmotions is provided to be fast enough for an operator to be convenientyet safe. Movement of the robotic arm 506 is controlled during surgeryto be meticulous and accurate. In addition, movement of the robotic armis controlled to be smooth and predictable through the entire range ofmotion required for a surgical procedure. As described herein, movementof the robotic arm 506 is controllable by remote control or via manualmanipulation of the arm itself. In some embodiments, the robotic arm 506is configured to be positionable with minimal force (e.g., via anassisted guidance feature) with just the use of, for example, a singleauricular finger.

In some embodiments, the robotic arm 506 may include mechanically orelectronically locking brakes on the joints. The brakes may be engagedonce the aim or “pose”, which is generally the location and direction,of the camera 300 after it is set by an operator. The robotic arm 506may include a locking or unlocking switch or other input device toprevent undesired manual or accidental motion. When locked, the examplerobotic arm provides sufficient stability that enables the stereoscopicvisualization camera 300 to provide a stable, clear image. The roboticarm 506 may additionally or alternatively include one or more dampeningdevices to absorb or attenuate vibrations following movement of thestereoscopic visualization camera 300 to a new pose. The dampeningdevices may include, for example, fluid-filled linear or rotationaldampeners, rubber-based vibration isolation mounting dampeners, and/ortuned mass-spring dampeners. Alternatively, or in addition, the arm 506may include electromechanical dampening, for example, through the use ofa proportional integral derivative (“PID”) servo system.

The example robotic arm 506 may be configured with a stowage position towhich one or more links are returned for transportation and storage. Astowage position enables the robotic arm to be transported and stored ina concise footprint yet deployed with a long reach required in somesurgical procedures. Cables, such as those routed for the stereoscopicvisualization camera 300, are provided along the robotic arm 506 so asto avoid interference with a surgical procedure.

In the illustrated embodiment of FIG. 34, the robotic arm 506 includessix joints, labeled R1, R2, R3, R4, R5, and R6. In other embodiments,the robotic arm 506 may include fewer or additional joints.Additionally, in some embodiments, at least some of the joints R1 to R6have rotational motion capabilities of +/−360°. The rotational motionmay be provided by an electromechanical subsystem that includes, foreach joint, an electric motor configured to drive a mechanicalrotational joint through one or more anti-backlash joint gearboxes. Eachof the joints R1 to R6 may include one or more rotational sensors todetect joint position. Further, each joint may include a slip clutchand/or an electromechanical brake.

Each of the joints R1 to R6 may have an overall repeatability of motion(with the camera 300 attached) of approximately +/− 1/10 of a millimeter(“mm”). The joints may be have variable rotational speeds that can becontrolled between 0.5° to 180° per second. Together, this translates tocamera movement between 1 mm per second to 1 meter per second. In someembodiments, the stereoscopic robotic platform 516 may have speedgovernors for one or more of the joints R1 to R6 that are in placeduring surgical procedures. Each of the joints R1 to R6 may beelectrically connected to a power source and/or command line in acontroller of the robotic arm 506. Wires for power and command signalsmay be routed internally within the joints and links. Further, one ormore of the joints may include dampeners, such as o-rings for connectionto links. The dampeners may, for example, reduce or absorb vibrations inthe robotic arm 506, vibrations from the cart 510, and/or vibrationsimparted via the stereoscopic visualization camera 300.

Joint R1 includes a base joint that is mechanically coupled to a flange3402, which is secured to a stationary structure 3404. The flange 3402may include any type of mechanical connector. The stationary structure3404 may include, for example, the cart 510 of FIG. 5, a wall, aceiling, a table, etc. The joint R1 is configured to rotate around afirst axis 3410, which may include the z-axis.

Joint R1 is connected to joint R2 via a link 3430. The example link 3430includes a cylinder or other tubular structure configured to providestructural support for the downstream sections of the robotic arm 506.The link 3430 is configured to provide a rotational secure connectionwith joint R2 to enable joint R2 to rotate while the link 3430 is heldin place by its connection to the joint R1. Joint R2 may include, forexample, a shoulder joint configured to rotate around an axis 3412. Theexample axis 3412 is configured to be perpendicular (or substantiallyperpendicular) to axis 3410. The axis 3412 is configured to be within anx-y plane given the rotation of the joint R1 around the z-axis.

Joint R2 is mechanically coupled to joint R3 via link 3432. The link3432 is configured to have a greater length than the link 3430 and isconfigured to provide structural support for downstream portions of therobotic arm 506. Joint R3 may include, for example, an elbow joint.Together with joint R2, joint R3 provides extensible positioning and/ororientating of the robotic arm 506. The joint R3 is configured to rotatearound an axis 3414, which is perpendicular or orthogonal to the axis3410 and parallel to the axis 3412.

Joint R3 is connected to joint R4 via link 3434, which providesstructural support for downstream portions of the robotic arm 506. Theexample joint R4 may be, for example, a first wrist joint configured toprovide rotation around axis 3416, which may be orthogonal to the axes3412 and 3414. Joint R4 is mechanically connected to joint R5 via link3436. Joint R5 may be a second wrist joint configured to providerotation around an axis 3418, which is orthogonal to axis 3416. Joint R5is mechanically connected to joint R6 via link 3438. Joint R6 may be athird wrist joint configured to rotate around axis 3420, which isorthogonal to the axis 3418. Together, the wrist joints R4 to R6 provideprecise flexibility in positioning the stereoscopic visualization camera300 described herein.

The example robotic arm 506 includes a connector 3450. The exampleconnector 3450 is connected to joint R6 via link 3440. In someembodiments, the example link 3440 may include a sleeve that enablesjoint R6 to rotate the connector 3450. As discussed herein, theconnector 3450 may be configured to mechanically couple to the couplingplate 3304 or the stereoscopic visualization camera 300 directly when acoupling plate is not used. The connector 3450 may include one or morescrews to secure the robotic arm 506 to the coupling plate 3304 and/orthe stereoscopic visualization camera 300.

In some embodiments, the robotic arm 506 of the illustrated example mayhave a maximum reach of 85 mm, in an orientation roughly similar to ahuman arm. The arm 506 may have a payload capacity of 5 kilograms.Further, the arm 506 may be configured as a “collaborative” device toenable safe operation in the proximity of humans. For example, themaximum force that the robotic arm 506 can apply to external surfaces iscontrolled. Should a portion of the robot arm unexpectedly contactanother object, the collision is detected, and motion is instantlyceased. During an emergency stop situation where for example, power islost the joints R1 to R6 can be back-driven or manually rotated suchthat an operator can grab part of the robotic system and swing it out ofthe way. For example, slip clutches within the joints limit the maximumtorque the joint motor can apply rotationally to the arm 506 duringoperation. When powered off, the slip clutches of the joints slip whenmanually manipulated to allow an operator to quickly move the roboticarm 506 out of the way.

FIGS. 35 to 40 illustrate example configurations of the robotic arm 506and the stereoscopic visualization camera 300, according to exampleembodiments of the present disclosure. FIG. 35 shows a diagram of therobotic arm 506 connected to the cart 510 via the flange 3402. In thisexample, the stereoscopic visualization camera 300 is connected directlyto the connector 3540. In this embodiment, the connector 3540 and/or thestereoscopic visualization camera 300 may include the sensor 3306 ofFIG. 33 for sensing translational and/or rotational force/motionimparted by an operator on the stereoscopic visualization camera 300. Ifthe connector 3540 includes the sensor 3306, the output force/motiondata may be transmitted through the robotic arm 506 to a controller. If,for example, the sensor 3306 is located on the stereoscopicvisualization camera 300, the output data may be transmitted withcontrol data to a separate controller. In some embodiments, a controllermay be provided in the cart 510 or separately at a server.

FIG. 36 shows an embodiment where the robotic arm 506 is mounted to aceiling plate 3404 via the flange 3402. The robotic arm may be suspendedfrom the ceiling of an operating room to reduce floor space clutter. Therobotic arm 506, including the joints, can be positioned above andtraversed from the area where surgical activity is being performed, outof the way of the surgeon and operating room staff, yet still providingfunctional positioning and/or orientating of the camera 300 whileproviding a clear view of the display monitors 512 and 514.

FIG. 37 shows an embodiment of the coupling plate 3304. In theillustrated example, a first end 3702 of the coupling plate 3304 isconnected to the connector 3450 of the robotic arm 506. A second end3704 of the coupling plate 3304 is connected to the stereoscopicvisualization camera 300. The example coupling plate 3304 is configuredto provide additional degrees of freedom for moving the stereoscopicvisualization camera 300. The coupling plate 3304 also extends themaximum reach of the robotic arm 506. The coupling plate 3304 may have alength between 10 cm to 100 cm.

The coupling plate 3304 may include one or more joints. In theillustrated example, the coupling plate 3304 includes joints R7, R8, andR9. The example joints are mechanical joints that provide rotationaround respective axes. The joints R7 to R9 may comprise rotatablelatching mechanisms that are movable after an operator actuates arelease button or lever. Each joint R7 to R9 may have its own releasebutton, or a single button may release each of the joints R7 to R9.

The joints R7 to R9 may be connected together via respective links. Inaddition, a link 3718 is provided for connection to the connector 3450of the robotic arm 506. Joint R7 is configured to rotate around axis3710, while joint R8 is configured to rotate around axis 3712, and jointR9 is configured to rotate around axis 3714. The axes 3710 and 3714 areparallel with each other and orthogonal to the axis 3712. Joints R7 andR9 may be configured to provide +/−360° rotation. In other examples,joints R7 and R9 may provide +/−90°, +/−180° rotation or +/−270°rotation around the respective axes 3710 and 3714. Joint R8 may provide+/−90° rotation around the axis 3712. In some examples, joint R8 mayonly be set at +90°, 0°, and −90°.

In some embodiments, joints R7 to R9 may include motors that providecontinuous movement. Joints R7 to R9 may also include control devices,such as switches or position sensors that communicate or provide dataindicative or a rotational position. In this manner, the joints R7 to R9may be similar to the joints R1 to R6 of the robotic arm 506 and providefor assisted movement and positioning sensing for feedback control.Power and control for joints R7 to R9 may be provided via wires routedthrough the robotic arm 506, power/wire connectors within the connector3450, and/or wires external to the robotic arm 506.

FIG. 37 shows an example where the stereoscopic visualization camera 300is positioned in a horizontal orientation such that an optical axis 3720is provided along a z-axis. The horizontal orientation may be used forimaging patients that are lying down. In contrast, FIG. 38 shows anembodiment where joint R8 is rotated by 90° to position the camera 300in a vertical orientation such that the optical axis 3720 is providedalong an x-axis or y-axis that is orthogonal to the x-axis. The verticalorientation may be used for imaging patients that are sitting. It shouldbe appreciated that joint R8 enables the stereoscopic visualizationcamera 300 to quickly be re-orientated between horizontal and verticalpositions based on the procedure.

In the illustrated examples of FIGS. 36 and 37, the example sensor 3306may be located at, for example, the connector 3450 of the robotic arm(with the connection of the coupling plate 3304) and/or at the first end3702 of the coupling plate (at the connection with the connector 3450).Alternatively or additionally, the example sensor 3306 may be locatedat, for example, the second end 3704 of the coupling plate (at theconnection with the camera 300) and/or at the camera 300 at theconnection with the second end 3704 of the coupling plate 3304.

FIGS. 39 and 40 show the stereoscopic visualization camera 300 in thehorizontal orientation and rotated +90° around the axis 3714 of jointR9. FIG. 40 shows an example of the stereoscopic visualization camera300 in the horizontal orientation and rotated −90° around the axis 3714of joint R9.

As illustrated in FIGS. 34 to 40, the example robotic arm 506 isconfigured to provide support for the stereoscopic visualization camera300 and allow for precise positioning and/or orientating and aiming ofthe camera's optical axis. Since the stereoscopic visualization camera300 does not have oculars and does not need to be oriented for asurgeon's eyes, there are many desirable positions and/or orientationsfor imaging that may be achieved that were not previously practical. Asurgeon can perform with the view most optimal for a procedure ratherthan that most optimal for his orientation to the oculars.

The example robotic arm 506, when used with the stereoscopicvisualization camera 300, enables a surgeon to see around corners andother locations that are not readily visible. The robotic arm 506 alsoenables patients to be placed into different positions including supine,prone, sitting, semi-sitting, etc. Accordingly, the robotic arm 506enables the patient to be placed in the best position for a specificprocedure. The example robotic arm 506, when used with the stereoscopicvisualization camera 300 can be installed for the least obtrusiveposition. The arm 506 and camera 300 accordingly provide a surgeonnumerous possibilities for visual locations and orientations while beingconveniently located and oriented out of the way.

The arrangement of the links and joints of the robotic arm 506 and/orthe coupling plate 3304, along with the motorized six (or nine) degreesof freedom generally allow the camera 300 to be positioned as desiredwith the link and joint configuration not unique to the pose of thecamera. As discussed in more detail below, the joints and links of thearm 506 and/or the plate 3304 may be manually repositioned and/orreoriented without changing the pose or FOV of the camera 300. Thisconfiguration allows, for example, an elbow joint to be moved out of anoccluding line of sight without changing the view of the surgical sitethrough the camera 300. Further, a control system can determine thelocation and pose of the camera 300 and calculate and displayalternative positions and/or orientations of the robotic arm 506 to, forexample, avoid personnel or display occlusion. Use of the variouspositions and/or orientations of the coupling plate 3304 along with anability of an image processor to flip, invert, or otherwise reorient thedisplayed image permit even more robot arm 506 positions and/ororientations.

The robotic arm 506 and/or coupling plate 3304 is/are generallysituated, and the joints are positioned such that joint singularitiesare avoided in any general movement. The avoidance of jointsingularities provides better robotic control of hysteresis andbacklash. Further, the lengths and configurations of the links andjoints of the robotic arm 506 and/or the coupling plate 3304 provide forsmooth movement along most any desirable motion paths. For example,repositioning and/or reorienting of the robotic arm 506 enables it tochange the direction of the camera 300 view of a target point within asurgical site without changing a focal point, thereby permitting asurgeon to view the same target point from differentdirections/orientations. In another example, the robotic arm 506 iscapable of changing a working distance to a target point withoutchanging a focal point by translating the camera 300 along the line ofsight towards or away from the target point. Numerous similar motionpaths are attainable as desired using the robotic arm 506 and/or thecoupling plate 3304 with the stereoscopic visualization camera 300 ofthe stereoscopic robotic platform 516.

B. Robotic Control Embodiment

The example robotic arm 506 and/or the coupling plate 3304 of FIGS. 34to 40 may be controlled by one or more controllers. FIG. 41 illustratesan embodiment of the stereoscopic robotic platform 516 of FIGS. 3 to 40,according to an example embodiment of the present disclosure. Theexample stereoscopic robotic platform 516 includes the stereoscopicvisualization camera 300 and corresponding image capture module 1404 andmotor and lighting module 1406 described in connection with FIGS. 14 and15.

In the illustrated embodiment, the stereoscopic robotic platform 516includes a server or processor 4102 that is located remote from thestereoscopic visualization camera 300. The processor 4102 may include,for example, a laptop computer, a workstation, a desktop computer, atablet computer, a smartphone, etc. configured with one or more softwareprograms defined by instructions stored in the memory 1570 that, whenexecuted by the processor 4102, cause the processor 4102 to perform theoperations described here. The example processor 4102 in this example isconfigured to include (or perform the operations described in connectionwith) the information processor module 1408, the image sensor controller1502, and/or the motor and lighting controller 1520 of FIGS. 14 to 16.

In some examples, at least some of the operations of the image sensorcontroller 1502, and/or the motor and lighting controller 1520 may beshared with the image capture module 1404 and motor and lighting module1406, respectively. For example, the processor 4102 may generatecommands for changing focus, magnification, and/or working distance, andvia a first portion of the motor and lighting controller 1520, and asecond portion of the motor and lighting controller 1520 within themotor and lighting module 1406 controls the drivers 1534 to 1552.Additionally or alternatively, a first portion of the informationprocessor module 1408 located operationally in the processor 4102 isconfigured to receive individual left/right images and/or stereoscopicimages from a second portion of the information processor module 1408 inthe image capture module 1404. The first portion of the informationprocessor module 1408 may be configured for processing the images fordisplay on one or more display monitors 512 and/or 514 including,visually fusing images with graphical guidelines/text, image overlaysfrom a MM machine, X-ray, or other imaging device, and/or fluorescenceimages.

The processor 4102 is electrically and/or communicatively coupled to theimage capture module 1404 and motor and lighting module 1406 of thestereoscopic visualization camera 300 via a wire harness 4102. In someembodiments, the harness 4102 may be external to the robotic arm 506. Inother embodiments, the wire harness 4102 may be internal or routedthrough the robotic arm. In yet other embodiments, the image capturemodule 1404 and motor and lighting module 1406 may communicatewirelessly with the processor 4102 via Bluetooth®, for example.

The example processor 4102 is also electrically and/or communicativelycoupled to the sensor 3306 via the wire harness 4102. The processor 4102is configured to receive, for example, rotational and/or translationaloutput data from the sensor 3306. The data may include digital dataand/or analog signals. In some embodiments, the processor 4102 receivesa near-continuous stream of output data from the sensor 3306 indicativeof detected force and/or motion. In other examples, the processor 4102receives output data at periodic sampled intervals. In yet otherexamples, the processor 4102 periodically transmits a request messagerequesting the output data.

In the illustrated example, the processor 4102 is furthercommunicatively coupled to at least one of a display monitor 512, inputdevices 1410 a, 1410 b, and other devices/systems 4104 (e.g., medicalimaging devices such as an X-ray machine, a computed tomography (“CT”)machine, a magnetic resonance imaging (“MRI”) machine, a camera aworkstation for storing images or surgical guidelines, etc.). The inputdevice 1410 a may include a touch screen device and the input device1410 b may include a foot switch. The touch screen input device 1410 amay be integrated with the display monitor 512 and/or provided as aseparate device on, for example, the cart 510 of FIG. 5. The exampledisplay monitor 512 is configured to display one or more user interfacesthat include a stereoscopic video (or separate two-dimensional left andright videos) of a target surgical site recorded by the stereoscopicvisualization camera 300.

The touch screen input device 1410 a is configured to provide one ormore user interfaces for receiving user inputs related to the control ofthe stereoscopic visualization camera 300, the coupling plate 3304,and/or the robotic arm 506. The input device 1410 a may include one ormore graphical control buttons, sliders, etc. that are configured toenable an operator to specify, set, or otherwise provide instructionsfor controlling a working distance, focus, magnification, source andlevel of illumination, filters, and/or digital zoom of the stereoscopicvisualization camera 300. The input device 1410 a may also include oneor more control buttons to enable an operator to select surgicalguidance graphics/text, a video and/or an image for fusing and/orotherwise superimposing on the displayed stereoscopic video displayed onthe display monitor 512. The input device 1410 a may also include a userinterface that is configured to enable an operator input or create asurgical procedure visualization template. The input device 1410 a mayfurther include one or more control buttons for controlling the roboticarm 506 and/or the coupling plate 3304, including options forcontrolling operational parameters such as speed, motion,deployment/stowing, calibration, target-lock, storing a view position,and/or changing or inputting a new orientation of the camera 300. Theuser interface controls for the robotic arm 506 and/or the couplingplate 3304 may include controls for moving the camera 300, which aretranslated into commands for the individual joints R1 to R9.Additionally or alternatively, the user interface controls for therobotic arm 506 and/or the coupling plate 3304 may include controls formoving each of joints R1 to R9 individually. Inputs received via theinput device 1410 a are transmitted to the processor 4102 forprocessing.

The example foot switch input device 1410 may include, for example, afood pedal configured to receive inputs for controlling a position ofthe stereoscopic visualization camera 300, the coupling plate 3304,and/or the robotic arm 506. For example, the foot plate input device1410 b may include controls for moving the camera 300 along the x-axis,the y-axis, and/or the z-axis. The foot plate input device 1410 b mayalso include controls for storing a position of the camera 300 and/orreturning to a previously stored position. The foot plate input device1410 b may further include controls for changing a focus, zoom,magnification, etc. of the camera 300.

In other embodiments, the stereoscopic robotic platform 516 may includeadditional and/or alternative input devices 1410, such as a joystick,mouse, or other similar 2D or 3D manual input device. The input devices1410 are configured to provide inputs similar to an X-Y panning device,with additional degrees of freedom resulting in flexibility of systemmotion. Input devices with 3D capabilities, such as a 3D mouse orsix-degree of freedom controller are well suited for flexible andconvenient input commands. A major benefit of these user control devicesis that the surgical image can be easily viewed while the motion isoccurring. Further, a surgeon can view what is happening around theentire surgical and nearby sites to avoid, for example, bumping thecamera 300 into surgical staff and/or nearby equipment.

Optionally, the input device 1410 may include a head, eye, orglasses-mounted tracking device, a voice recognition device, and/or agesture input device. These types of input devices 1410 facilitate“hands-free” operability such that an operator does not need to touchanything with their sterile gloves. A gesture-recognizing control may beused, where certain operation hand motions are recognized and translatedinto control signals for the camera 300, the coupling plate 3304, and/orthe robotic arm 506. A similar function is provided by avoice-recognition device, where a microphone senses a command from anoperator, such as “move the camera left”, recognizes the speech as acommand, and converts it into appropriate camera and/or robot controlsignals. Alternate embodiments include an eye tracking device that isconfigured to determine a position of an operator's eyes with respect toa 3D display, and can adjust the view depending on where in thedisplayed scene the operator is looking.

Other embodiments include a device configured to track a position of anoperator's head (via for example a trackable target or set of targetsthat are mounted on an operator's 3D glasses) in a frame of referenceand a footswitch to activate “head tracking”. The example tracking inputdevice is configured to store a starting position of an operator's headat activation time and then detects head position continually at someshort time interval. The tracking input device in conjunction with theprocessor 4102 may calculate a movement delta vector between a currentposition and the starting position and convert the vector intocorresponding robotic arm or camera lens movements. For example, atracking input device 1410 and the processor 4102 may convert left/righthead movements into robotic arm movements such that an image onscreenmoves left/right. The tracking input device 1410 and the processor 4102may also convert up/down head movements into robotic arm or camera lensmovements such that an image onscreen moves up/down, and may convertforward/back head movements into robotic arm or camera lens movementssuch that an image onscreen zooms in/out. Other movement conversions arepossible, for example, by converting head rotation into a“lock-to-target” motion of the robotic arm 506. As described here,lock-to-target is configured to maintain a focal point of the roboticplatform 516 on the same point in a scene or FOV to within sometolerance and pivot the robotic arm 506 (and hence the view) in adirection which mimics the head movement of an operator.

Prior to certain surgical procedures, a surgical plan is created thatestablishes desired paths for instruments and visualization. In someembodiments, the input device 1410 is configured to follow such apredetermined path with little further input from an operator. As such,the operator can continue operating while the view of the surgical siteis automatically changing as pre-planned. In some embodiments, thesurgical plan may include a set of pre-planned waypoints, correspondingto camera positions, magnification, focus, etc. An operator may actuatethe input device 1410 to progress through the waypoints (causing theprocessor 4102 to move the robotic arm 506, the coupling plate 3304,and/or the camera 300 as planned) as the surgical procedure progresses.

In the illustrated embodiment, the example sensor 3306 is an inputdevice. The sensor 3306 is configured to detect an operator's movementor force on the stereoscopic visualization camera 300 and convert thedetected force/movement into rotational and/or translational data. Thesensor 3306 may include a motion-anticipation input device, such as asix-degrees-of-freedom haptic force-sensing module or an opto-sensor(e.g., force/torque sensor), that enables the robotic arm 506 to respondelectromechanically to an operator's gentle push on the camera 300. Theopto-sensor may include an electro-optical device configured totransform applied forces and/or torques into electrical signals, therebyenabling a desired force/torque input by an operator to be sensed andtransformed into a motion request that is provided in the sensed linearand/or rotational direction(s). In other embodiments, other sensorstypes may be used for the sensor 3306. For example, the sensor 3306 mayinclude a strain gauge or piezoelectric device that is configured tosense a haptic request from an operator.

In an embodiment, a surgeon holds one or more of the control arms 304and actuates or pushes a release button (which may be located on one orboth of the control arms 304). Actuation of the release button causesthe camera 300 to transmit a message to the processor 4102 indicativethat an operator desires to begin an “assisted-movement” mode. Theprocessor 4102 configures the robotic arm 506 and/or the coupling plate3304 to enable the surgeon to gently steer the camera 300 in a desireddirection. During this movement, the processor 4102 causes the roboticarm 506 and/or the coupling plate to move the camera 300 in a “powersteering” manner, safely supporting its weight and automaticallydetermining which joints should be activated and which should be brakedin a coordinated manner to achieve the surgeon's desired movement.

In the illustrated example, the stereoscopic robotic platform 516 ofFIG. 41 includes a robotic arm controller 4106 that is configured tocontrol the robotic arm 506 and/or the coupling plate 3304. The roboticarm controller 4106 may include a processor, a server, amicrocontroller, a workstation, etc. configured to convert one or moremessages or instructions from the processor 4102 into one or moremessages and/or signals that cause any one of joints R1 to R9 to rotate.The robotic arm controller 4106 is also configured to receive andconvert sensor information, such as joint position and/or speed from therobotic arm 506 and/or the coupling plate 3304 into one or more messagesfor the processor 4102.

In some embodiments, the robotic arm controller 4106 is configured as astand-alone-module located between the processor 4102 and the roboticarm 506. In other embodiments, the robotic arm controller 4106 may beincluded within the robotic arm 506. In yet other embodiments, therobotic arm controller 4106 may be included with the processor 4102.

The example robotic arm controller 4106 includes one or moreinstructions stored in a memory 4120 that are executable by a roboticprocessor 4122. The instructions may be configured into one or moresoftware programs, algorithms, and/or routines. The memory 4120 mayinclude any type of volatile or non-volatile memory. The example roboticprocessor 4122 is communicatively coupled to the processor 4102 and isconfigured to receive one or more messages related to operation of therobotic arm 506 and/or the coupling plate 3304. The example roboticprocessor 4120 is also configured to transmit to the processor 4102 oneor more messages that are indicative of positions and/or speeds ofjoints R1 to R9. The one or more messages may also be indicative that ajoint has reached a travel-stop or is being prevented from moving.

The example processor 4120 is configured to determine which joints R1 toR9 are powered in a coordinated manner such that a totality of allmotions of all the joints results in the desired image motion at thecamera 300. In a “move the camera left” example there may be complexmotions of several joints which cause the camera's surgical image toappear to simply and smoothly translate to the left, from a relativeviewpoint of a surgeon. It should be noted that in the “move the cameraleft” example, depending on how the camera 300 is connected to therobotic arm 506 through the coupling plate 3304, the control signals tospecific joints may be drastically different depending on theposition/orientation.

The memory 4120 may include one or more instructions that specify howjoints R1 to R9 are moved based on a known position of the joints. Therobotic arm controller 4106 is configured to execute the one or moreinstructions to determine how instructed camera movement is translatedinto joint movement. In an example, the robotic arm controller 4106 mayreceive messages from the processor 4102 indicative that thestereoscopic visualization camera 300 is to move downward along a z-axisand move sideward in an x-y plane. In other words, the processor 4102transmits indicative of inputs received via the input devices 1410regarding desired movement of the camera 300. The example robotic armcontroller 4106 is configured to translate the vectors of movement in3-dimensional coordinates into joint position movement information thatachieves the desired position/orientation. The robotic arm controller4106 may determine or take into account the current location of thelinks and joints of the robotic arm 506 and/or the coupling plate 3304(and/or a position/orientation of the camera 300) in conjunction withthe desired movement to determine a movement delta vector. In addition,the robotic arm controller 4106 may perform one or more checks to ensurethe desired movement does not cause the camera 300 to enter into orprogress close to a restricted area, as specified by one or morethree-dimensional boundaries that are defined in the same coordinatesystem as the arm 506 and coupling plate 3304. Areas close to a boundarymay specify a reduced scale factor that is applied by the robotic armcontroller 4106 when movement signals are sent to the joints, whichcauses the joints to move slower as the robotic arm 506 approaches aboundary, and not move any further past a boundary.

After the boundary checks are performed, the robotic arm controller 4106uses the movement delta and the current position/orientation of each ofjoints R1 to R9 to determine an optimal or near optimal movementsequence for rotating one or more of the joints to cause the robotic arm506 to move the camera 300 into the specified location. The robotic armcontroller 4106 may use, for example, an optimization routine thatdetermines a minimal amount of joint movement needed to satisfy themovement delta vector. After the amount of joint movement is determined,the example robotic arm controller 4106 is configured to send one ormore messages (indicative of an amount of rotation and speed ofrotation, taking into account any scale factors) to a motor controller4124. The robotic arm controller 4106 may transmit a sequence ofmessages to cause the robotic arm 506 and/or coupling plate 3304 to movein a defined or coordinated sequence. The sequence of messages may alsocause a change in joint speed as, for example, the robotic arm 506approaches a virtual or physical boundary.

The example motor controller 4124 is configured to translate or covertthe received messages into analog signals, such as pulse-width modulated(“PWM”) signals that cause one or more of joints R1 to R9 to rotate. Themotor controller 4124 may, for example, select the input line to theappropriate joint motor, where a pulse duration is used for controllinga duration of time that the motor rotates and a frequency, duty cycle,and/or amplitude of the pulse is used to control rotation speed. Themotor controller 4124 may also provide power for the joint motors andcorresponding joint sensors.

In some embodiments, the robotic arm controller 4106 in combination withthe motor controller 4124 is configured to receive or read joint sensorposition information and determine, through kinematics, the location andorientation of the robotic joints and camera 300. Each joint R1 to R9may include at least one sensor that detects and transmits dataindicative of joint position, joint rotational speed, and/or jointrotational direction. In some embodiments, the sensors transmit onlyposition information, and speed/direction are determined by the roboticarm controller 4106 based on differences in the position informationover time. The robotic arm controller 4106 may transmit the sensor datato the processor 4102 for determining movement information.

The robotic arm controller 4106 receives movement instructions from theprocessor 4102 and determines, through Jacobian, forward, and/or inversekinematics, which motors and joints should be activated, how fast andhow far, and in what direction. The robotic arm controller 4106 thensends the appropriate command signals to motor power amplifiers in themotor controller 4124 to drive the joint motors in the robotic arm 506.

The example robotic arm 506 receives appropriate motor power signals andmoves accordingly. Sensors and brakes in the arm 506 react to thevarious operations and feedback information from the robotic armcontroller 4106. In some embodiments, the robotic arm 506 ismechanically and communicatively connected to the coupling plate 3304,which transmits coupler status and orientation information to therobotic arm controller 4106.

In some embodiments, the example robotic arm 506 of FIG. 41 includes acoupler controller 4130. The example coupler controller 4130 isconfigured to bypass the robotic processor 4106 and relay controlinformation between the processor 4102 and the coupling plate 3304. Thecoupler controller 4130 may receive messages from the processor 4102 andcorrespondingly cause joints R7 to R9 rotate on the coupling plate 3304.The coupler controller 4130 may also receive sensor informationregarding joint position and/or speed and transmit one or more messagesto the processor 4102 indicative of the joint position and/or speed. Inthese embodiments, the processor 4102 may transmit messages forcontrolling the robotic arm 506 and separate messages for the couplingplate 3304.

In some embodiments, the robotic arm controller 4106 is configured todetermine how joints R7 to R9 are to move. However, if the couplingplate 3304 is not communicatively coupled directly to the robotic arm506, the robotic processor 4106 may transmit the movement signals to thecoupler controller 4130 via the processor 4102. In instances where atleast some operators of the robotic processor 4106 are located with theprocessor 4102, the coupler controller 4130 receives movement commandsor signals from the processor 4102 in conjunction with the robotic arm506 receiving movement commands or signals from the processor 4102.

In the illustrated embodiment of FIG. 41, the example stereoscopicvisualization camera 300, the processor 4102, the coupling plate 3304,the robotic arm 506, the robotic arm controller 4106, and/or the inputdevices 1410 receive power via an input power module 4140. The examplemodule 4140 includes a power supply (such as power from a wall outlet)and/or an isolation transformer to prevent powerline anomalies fromdisrupting system performance. In some instances, the power supply caninclude a battery power supply.

The stereoscopic visualization platform 516 may also include anemergency stop switch 4142 that is configured to immediately cut offpower. The switch 4142 may only cutoff power to the robotic arm 506and/or the coupling plate 3304. The processor 4106 may detect activationof the emergency stop switch 4142 and cause joint brakes to engage toprevent the robotic arm 506 from falling. In some instances, the roboticarm 506 is configured to activate the joint brakes after detecting aloss of power. In some embodiments, joints R1 to R6 of the robotic arm506 are configured to slip if a force above a threshold is applied,thereby enabling an operator to quickly move the arm out of the way inan emergency, with or without power.

In some embodiments, the example processor 4102 is configured to displayone or more graphical representations of the robotic arm 506, thecoupling plate 3304, and/or the stereoscopic visualization camera 300.The processor 4102 may cause the graphical representation to bedisplayed in one or more user interfaces that provide control for therobotic arm 506, the coupling plate 3304, and/or the camera 300. Theprocessor 4102 may position and orient the graphical representation toreflect the current position of the robotic arm 506, the coupling plate3304, and/or the camera. The processor 4102 uses, for example, feedbackmessages from the robotic arm controller 4106 to determine which jointsin the graphical representation are to be rotated, thereby changing theorientation and/or position of the display device. In some instances,the processor 4102 is configured to receive user input via the graphicalrepresentation by, for example, an operator moving the links, joints, orcamera 300 in the graphical representation to a desired position. In thecase of movement of the stereoscopic visualization camera 300, theprocessor 4102 may transmit the new coordinates corresponding to wherethe camera was moved. In the case of moved joints or links, theprocessor 4102 may transmit to the robotic arm controller 4106 messagesindicative of joint rotation and/or positions of links.

In some embodiments, the processor 4102 operates in connection with therobotic arm controller 4106 to adjust one or more lenses of the camerabased on or in cooperation with movement of the robotic arm 506 and/orthe coupling plate 3304. For example, if the robotic arm 506 is movedtoward a surgical site, the processor 4102 operates in connection withthe robotic arm controller 4106 to change a working distance or focalpoint by moving one or more of the lenses of the stereoscopicvisualization camera 300 to maintain focus. The processor 4102 operatesin connection with the robotic arm controller 4106 to determine, forexample, that movement of the robotic arm 506 causes a working distanceto decrease. The EPU processor 4102 operates in connection with therobotic arm controller 4106 to determine a new position for the lensesbased on the new working distance set by moving the robotic arm 506.This may include moving one or more lenses for adjusting focus. In someembodiments, the processor 4102 may instruct the camera 300 to operate acalibration routine for the new position of the robotic arm 506 toeliminate, for example, spurious parallax.

In some instances, an operator may be changing positions of one or morelenses of the stereoscopic visualization camera 300 and reach a lenstravel limit. The position of the lenses is sent from the camera 300 tothe processor 4102, which to determine that a limit has been reached.After detecting that a limit has been reached, the processor 4102 maycause the robotic arm 506 to move (via the controller 4106) based oninput from the operator, thereby extending their command from lensmovement to arm movement to reach a desired magnification or targetarea. As such, the processor 4102 operating in connection with therobotic arm controller 4106 enables an operator to use only one userinterface rather than changing between an interface for the robotic armand the camera. It should be appreciated that the processor 4102 and/orthe controller 4106 may check desired movement against any predeterminedmovement limits to ensure the movement will not cause the camera 300 orrobotic arm 506 to enter into restricted patient or operator space. If alimit violation is detected, the processor 4102 in connection with therobotic arm controller 4106 may display an alert to the operator (via auser interface displayed on the touchscreen input device 1410 a and/orthe display monitor 512) indicative of the limit to indicate a reasonthe robotic arm 506 was stopped.

C. Robotic Arm and Stereoscopic Camera Calibration Embodiment

As discussed above, the example stereoscopic visualization camera 300 isconfigured to provide high-resolution stereoscopic video images of atarget surgical site at different magnifications. As part of thestereoscopic visualization platform 516, the stereoscopic visualizationcamera 300 operates in connection with the robotic arm 506 and/or thecoupling plate 3304 for precise and clear changes to image focus,working distance, magnification, etc. To accomplish the imageacquisition flexibility, the stereoscopic visualization platform 516 isconfigured to operate one or more calibration, initialization, and/orreset routines. In some embodiments, the stereoscopic visualizationcamera 300, the robotic arm 506, the coupling plate 3304, or moregenerally, the stereoscopic visualization platform 516 is calibratedduring manufacture and/or after installation. Calibration of the camera300 with the robotic arm 506 provides positioning information of thecamera 300 relative to the robotic arm 506 and operator space. Afterpower-up of the stereoscopic visualization platform 516, in someembodiments, the camera 300 and/or the robotic arm 506 is configured toperform further calibration/initialization to measure and verify alocation and orientation of the camera 300 at that time.

The example processor 4102 is configured to store results from thecalibration (e.g., calibration data), in for example, the memory 1570and/or the memory 4120 of FIG. 41. The calibration results may be storedto calibration registers and/or lookup tables (“LUTs”) in the memories1570 and/or 4120. The stored calibration data relates or maps optical,functional, and/or performance characteristics to attributes of thecamera 300, the robotic arm 506, and/or the coupling plate 3304 that areadjustable, measurable, and/or verifiable by an operator or by theprocessor 4102. For example, a working distance actuator motor encoderposition for the main objective assembly 702 (of FIG. 7) is mapped in aLUT to a working distance. In another example a zoom lens axial positionalong a linear encoder for the zoom lens assembly 716 is mapped in a LUTto the magnification level. For each of these examples, the exampleprocessor 4102 is configured to determine the proper level of an encodercharacteristic, adjust, and verify that the characteristic provides thespecified or desired working distance and/or magnification. In someembodiments, LUTs may be compound, where multiple performancecharacteristics are mapped to multiple platform 516 attributes foroverall control of all relevant aspects of the camera 300, the roboticarm 506, and/or the coupling plate 3304.

The combination of a robotic arm 506 and the example stereoscopicvisualization camera 300 provides highly accurate position, direction,and/or orientation information of the target view with respect to aframe of reference of the robotic arm 506. The following sectionsdescribe how the stereoscopic visualization camera 300 is calibrated todefine a visual tip. After a visual tip is determined, the stereoscopicvisualization camera 300 is registered to a frame of reference of therobotic arm 506 and/or the coupling plate 3306. Accordingly, aftercalibration and registration, a stereoscopic view of a surgical site isunified with the integrated control of the stereoscopic visualizationcamera 300 combined with the position, direction, and orientationcontrol of the robotic arm 506 and/or coupling plate 3304.

In some embodiments, the example processor 4102 of FIG. 41 is configuredto integrate a registration of stereoscopic visualization camera 300,including its visual tip, precisely with a position, direction, and/ororientation calibration of the robotic arm 506 to define a unifiedposition, direction, and/or orientation awareness of acquiredstereoscopic images and all points therewithin, with respect to aprescribed coordinate frame. The example processor 4102 is configured touse intrinsic visual imaging data from the stereoscopic visualizationcamera 300 to coordinate or direct the physical positioning and/ororientating of the robotic arm 506 to provide visualization, as desiredby an operator. In addition, such direction and coordination provided bythe processor 4102 is provided to maintain preferred characteristics ofthe visualization such as focus, working distance, pre-definedpositioning, orientating, etc.

In some embodiments, calibration of the stereoscopic visualizationcamera 300, the robotic arm 506, and/or the coupling plate 3304generally includes (i) determining and/or measuring inaccuracies offunctional parameters of the stereoscopic visualization platform 516that affect stereoscopic images; (ii) calibrating or adjusting, thestereoscopic visualization platform 516 to minimize the inaccuracies inthe stereoscopic images at or below a desired level; (iii) verifyingthat the adjustments have been made within a desired level ofcalibration accuracy through simultaneous comparisons of the dualchannels of the stereoscopic images to each other or calibrationtemplates; and (iv) using the stereoscopic visualization platform 516 inthe performance of its tasks, where a level of the accuracy of thecalibration is detectable and maintained.

In an alternative embodiment, the robotic arm 506 is provided with oneor more fixed calibration fiducials that are used to precisely calibratea physical relationship of the joint and links of the robotic arm 506 toone another as well as to calibrate a relationship of the visual tip ofthe camera 300 to the robotic arm 506 and/or an initial poseconfiguration. The robotic platform fixed calibration fiducials can beused to register or integrate the robotic arm 506 with an externalenvironment, such as an operating room theater or with a patient ortarget space within an external environment. The fixed calibrationfiducials can either include a dedicated attachment to an externalportion of a body of the robotic arm 506 or a combination of knownexternal features of the robotic arm 506, such as mounting points,joints, corners, or the like.

1. Calibration of the Stereoscopic Visualization Camera Embodiment

To match a stereoscopic view of a surgical site, the example processor4102 and/or the stereoscopic visualization camera 300 is configured toperform one or more calibration routines. The example routines may bespecified by one or more instructions stored in the memory 1570, thatwhen executed by the processor 4102, cause the processor 4102 todetermine lens position corresponding to certain working distances,magnifications (e.g., zoom level), and focus levels. The instructionsmay also cause the processor 4102 to operate one or more routines fordetermining a center of projection, stereo optical axis, and/or a ZRPfor the stereoscopic visualization camera 300 at different workingdistances and/or magnifications. Calibration enables, for example, theprocessor 4102 to retain focus on a target surgical site whenmagnification and/or working distance is changed.

FIG. 42 illustrates an example procedure 4200 or routine for calibratingthe stereoscopic visualization camera 300, according to an exampleembodiment of the present disclosure. Although the procedure 4200 isdescribed with reference to the flow diagram illustrated in FIG. 42, itshould be appreciated that many other methods of performing the stepsassociated with the procedure 4200 may be used. For example, the orderof many of the blocks may be changed, certain blocks may be combinedwith other blocks, and many of the blocks described are optional.Further, the actions described in procedure 4200 may be performed amongmultiple devices including, for example the optical elements 1402, theimage capture module 1404, the motor and lighting module 1406, and/orthe information processor module 1408 of the example stereoscopicvisualization camera 300. For example, the procedure 4200 may beperformed by one of the programs 1560 of the information processormodule 1408.

The example procedure 4200 begins when the stereoscopic visualizationcamera 300 is powered or otherwise initialized (block 4202). The camera300 may be mounted to the robotic arm 506. Alternatively, the procedure4200 can be performed when the stereoscopic visualization camera 300 isconnected to a fixed mount. The example procedure 4200 next performs ZRPalignment, as discussed above in connection with FIGS. 25 and 26 (block4204). The example processor 4102 may automatically align the ZRPs, asdiscussed above, and/or operate in connection with an operator toprovide alignment of the left and right optical paths on the imagesensors 746, 748. In some examples, the processor 4102 and/or anoperator may cause small movements or flexing of a flexure (e.g., theflexure 1300 of FIG. 13) via a motor with sufficient accuracy to makevery small adjustments to a tilt of a lens component to move a ZRP intoalignment with a pixel grid origin. During semi-manual alignment, theprocessor 4102 may cause left and right images from the image sensors746 and 748 to be overlaid on the display monitor 512. An operator mayuse the input device 1410 to adjust the images, causing pixel sets ofthe sensors 746 and 748 to be accordingly moved until the ZRPs areproperly aligned.

During alignment, the ZRPs are set to be aligned at an image center toavoid spurious parallax. Alignment to within about a single pixel on adisplay is possible. The degree of alignment from the left and rightviews to an image center is visible in the overlaid images, includingduring zooming operations. In an example of a 8° FOV, the use of a 4Kimage sensor 746, 748 and a corresponding 4K display resolution of thedisplay monitor 512 (comprising about 4000 pixels by about 2000 rows)produces a system resolution of 8°/4000 pixels=7 arc-seconds. However,ZRP alignment can be performed at most any magnification, where theresolution (being the same number of pixels (e.g. 4000)) is divided by areduced (or increased) angular FOV. For example, an exemplary embodimentof the camera 300 at a high magnification produces an angular FOV ofabout 2°. An 8K UHD display monitor 512 and sensors 746 and 748 haveabout 8000 pixels in about 4000 rows. The resolution of this system is2°/8000 pixels=1 arc-seconds. This is about an order of magnitude, ormore, better than known systems in the art, in which the accuracy ofassembled, individually-measured components with tolerances that haveresolutions measured in arc-minutes. As images sensors and displaymonitors become higher in density with generally smaller pixels in thesame physical sensor or display space, the accuracy of the stereoscopicvisualization camera 300 adjustability scales with the smaller pixelsize. The enhanced high accuracy alignment of the stereoscopicvisualization camera 300 provides for better, more accurate digitaleffects.

The alignment of the ZRPs is complete when the ZRPs of the left andright images remain at an image center within a desired tolerance rangeand the images of the target surgical site remain accurate when cycledfrom low magnification to high magnification. After the ZRPs are alignedthroughout the magnification capabilities of the stereoscopicvisualization camera 300, the pixel set locations and/or lens locationsfor each of the magnification levels are stored to, for example, a LUT4203 or other data structure. In other examples, the processor 4102writes, to calibration registers, the pixel set locations and/or lenslocations for each of the magnification levels.

After the ZRPs are aligned, the example processor 4102 is configured tocalibrate for working distance and/or magnification (e.g., zoom) (block4206). As discussed above in connection with the working distance andzoom examples of FIG. 15, precise knowledge of working distance isimportant in the camera 300 so that the robotic arm 506 may preciselyposition the camera relative to desired coordinates. In some instances,an accurate fiducial is used, along with mechanical dimensions of therobotic arm 506, to transform object plane data from an image into acoordinate system respective of the stereoscopic visualization platform516, referred to herein as robot space.

The example calibration procedure 4200 is performed to map the workingdistance of the optical system of the stereoscopic visualization camera300, where the working distance may be calculated or measured inmillimeters from a front face of a common mode objective (“CMO”) lensassembly (e.g., the front working distance main objective lens 408 ofFIGS. 4 and 7) to an object plane. The working distance is mapped to aknown measurable parameter, such as for example, a focus motor position,measured in counts of a motor shaft encoding device from a known “home”location such as a physical stop or limit switch trigger position.

The calibration at block 4206 is performed by the processor 4102sequentially moving the object plane in discrete steps along the opticalaxis and re-focusing the image while recording the encoder counts andthe working distance, as discussed in more detail in conjunction withFIG. 43. The processor 4102 measures the working distance externallyfrom the front face of the CMO. The mapping of the encoder counts andthe working distance is stored to the LUT 4203, or a different LUTand/or calibration registers. This calibration enables the processor4102 to output an encoder count position to the motor controller, givena desired working distance. Exemplary embodiments of the stereoscopicvisualization camera 300 use high-count-per-revolution shaft encodingdevices, where resolution of the working distance is on the order of 1micron per each encoder count. Alternative embodiments may includedifferent encoder resolution to provide higher or lower resolution ofworking distance, as desired.

FIG. 43 shows an embodiment of the example stereoscopic visualizationcamera 300 moving an object plane in discrete steps, according to anexample embodiment of the present disclosure. The example stereoscopicvisualization camera 300 includes the main objective assembly 702 ofFIG. 7 (e.g., a single CMO), which is configured to provide left andright views of a target surgical site. In the illustrated example, themain objective assembly 702 is shown as an achromatic refractiveassembly with the stationary front working distance lens 408 within ahousing 4302 and the movable rear working distance lens 740, which ismovable along the z-axis (or other optical axis). Movement of the rearworking distance 704 changes the distance to the front working distancelens 408. The spacing between the lenses 408 and 704 determines theoverall front focal length 4304 of the main objective assembly 702, andaccordingly the location of a front focal plane (or simply “focusplane”) 4306. The front focal plane 4306 is located at a distance equalto the focal length 4304 from a principal plane 4308 of the mainobjective assembly 702. It may be difficult to gauge the location of theprincipal plane 4308, so a distance from the bottom surface of housing4302 to the front focal plane is defined as the working distance 4310.The working distance 4310 accordingly accurately sets a plane of thetarget site or scene that is in focus.

Imaging an object at the front focal plane 4306 develops a conjugateimage located at infinity from a back or rear of the main objectiveassembly 702. Two parallel optical paths comprising optics and sensors714, 716, 718, 744R, and 744L of the camera 300, transversely separatedby an interpupillary distance (“IPD”) 4312, generate left and rightviews along respective left 4320 and right 4322 optical axes in slightlydifferent directions from an optical axis 4324 of the main objectiveassembly 702. The two optical paths are adjusted such that theirrespective external converging left and right axes are set to coincideat the center of the FOV of an image 4330. This point 4330 is referredto herein as the “tip” of the stereoscopic visualization camera 300 atthe front focal plane 4306.

Adjustment of a position of the rear working distance lens 740 causes achange in the front focal length 4304 main objective assembly 702. Asillustrated, a change in the position of the rear working distance lens740 creates a new working distance 4310′ that is located at the positionof a new front focal plane 4306′. The movement of the rear workingdistance lens 740 also causes a realignment of left 4320′ and right4322′ optical axes, resulting in a relocated tip 4330′ of the camera300. Visualization of an object with the camera 300 above or below thefocus plane 4306 diminishes a focus of the object.

In a manner similar to that for working distance calibration, a similarLUT, or additional columns in a working distance LUT 4203, can beconstructed by the processor 4102 varying the magnification whilemeasuring an image height of an object of known size. Magnification canbe quantified by determining the counts of the motor shaft encodingdevice from a known “home” location such as a physical stop or limitswitch trigger position. The processor 4102 can measure image heightrelatively, for example in a number of sensor pixels at eachmagnification position. Magnification can be characterized by, forexample dividing the height in pixels by the height in millimeters.

Returning to FIG. 42, after the working distance and magnification ofthe stereoscopic visualization camera 300 are calibrated, the exampleprocessor 4102 is configured to determine a center of projection (block4208). The center of projection (e.g., COP) may be determined using oneor more routines that model the stereoscopic visualization camera 300,as discussed above in connection with FIG. 15. To match the left andright stereoscopic view of a surgical site, it is often desirable tomodel the physical camera 300 using a mathematical model implemented insoftware, firmware, hardware, and/or GPU code for the processor 4102. Aperspective of a 3D computer model, such as the MM image of a braintumor, can often be rendered and viewed from user-adjustable directionsand distances (e.g. as if the images are captured by a synthesizedstereoscopic camera). The adjustability of the model may be used by theprocessor 4102 to match a perspective of a live surgical image, whichmust therefore be known.

Exemplary embodiments of the stereoscopic visualization camera 300and/or processor 4102 are configured to accurately measure and calculatecamera model parameters for each value of magnification and workingdistance. These values are controlled by separate optics containedwithin the stereoscopic visualization camera 300. The dual optics arealigned such that the parallax at the center of an image between theleft and right channels/views is approximately zero at the focal plane4330. Additionally, the stereoscopic visualization camera 300 isparfocal across the magnification range, and par central acrossmagnification and working distance ranges because the ZRPs of each leftand right channel have been aligned to the centers of their respectivepixel grids (described above in block 4202). In other words, changingonly the magnification keeps the image in focus in both channels, andtrained on the same center point. Similarly, changing only a workingdistance should cause no vertical parallax in the image, only increasedhorizontal parallax between the left and right views, if the target andstereoscopic visualization camera 300 remain stationary.

FIG. 44 illustrates a graph 4400 illustrative of a routine executable bythe processor 4102 for determining a COP of the stereoscopicvisualization camera 300, according to an example embodiment of thepresent disclosure. In the illustrated example, a COP of a pinhole ormodeled camera 300 is along an optical axis 4402 at the plane of thepinhole (O). To determine the COP for the camera model, a virtualpinhole camera model is used, where the processor 4102 is configured todetermine an actual focus distance 4404 from the COP to an object plane.During the calibration routine, the processor 4102 keeps themagnification of the camera 300 fixed while measurements are recorded ofan image height 4406, for example in the number of pixels at a plane ofthe optical image sensor 744, with an object of height 4408 at threedifferent distances along the optical axis 4402: at the object plane,and at a distance “d” less than the object plane distance, and at adistance “d” greater than the object plane distance. The processor 4102uses routines that include algebra based on similar triangles at the twomost extreme positions to determine the focus distance 4404 to a COP4410. The processor 4102 may determine focus distance at alternativemagnifications based on the ratio of the alternative magnification tothe magnification used for the calibration.

Returning to FIG. 42, the example processor 4102 is configured todetermine COPs for varying working distances and magnifications. Theprocessor 4102 relates motor shaft encoder counts for the lenses to theCOP for the varying working distances and magnifications in the LUT4203, a different LUT, or one or more calibration registers. In someembodiments, the processor 4102 may only store a relation of a COP forone magnification and/or working distance and calculate the othermagnifications and/or working distances using the one known COPrelation.

After calibrating for a COP, the example processor 4102 is configured tocalibrate stereoscopic left and right optical axes and an interpupillarydistance (“IPD”) between the axes of the stereoscopic visualizationcamera 300 (block 4210). To characterize the optics of the stereoscopicvisualization camera 300, the IPD between the left and rightchannels/views should be known. In embodiments, the IPD may be designedinto the mechanical components holding the sensors and optics shown inFIG. 7. IPD is thus set mechanically. However, the actual optical axismay differ from the mechanical axis of the optical elements and theirmounts. Other embodiments enable the IPD to be varied within thestereoscopic visualization camera 300.

In some applications, it is desirable to precisely know the direction ofthe stereoscopic optical axis with respect to a fiducial or mechanicalaxis on a frame of the stereoscopic visualization camera 300. Thisenables, for example, the processor 4102 to aim the stereoscopicvisualization camera 300 precisely through mechanical means. The aimingcan be characterized by a geometrically-defined view vector, looking outcoincidentally with the stereoscopic optical axis, with respect to aframe of reference of the stereoscopic visualization camera 300. Inaddition, clocking of the left and right channels for the optical sensor744 is included in a view vector, comprising the stereoscopicvisualization camera 300 orientation or pose.

FIG. 45 shows a plan view of an optical schematic that is illustrativeof how the IPD of the stereoscopic visualization camera 300 may bemeasured and calibrated, according to an example embodiment of thepresent disclosure. In the illustrated example, an optical axis 4502 isperfectly aligned with a mechanical axis 4504. The right image sensor746 and the left image sensor 748 (as approximated by one or more cameramodels) are spaced by an IPD 4506. The sensors 746 and 748 are alignedand focused on an object 4508 (in a target surgical site). The object4508 is placed at a focus distance 4510 from the sensors 746 and 748such that parallax at the plane of the object is theoretically zero, asdepicted in the display of the left or right view of object at focusplane 4512. In this exemplary example, for clarity the object 4508 is adisc, the front view of which is shown as item 4514.

FIG. 45 also illustrates another example where the object 4508 isdisplaced along the mechanical axis a distance “d” and is shown as item4508′. The displacement of the object 4508 generates parallax, whichappears in the displays of a left view 4520 as P_(L) and a right view4522 as PR. In this example, the mechanical and optical axes arecoincident and the parallax magnitudes are equal. The parallax can bemeasured, for example by counting a number of pixels of disparitybetween the left and right views 4520 and 4522 and multiplying by amagnification factor pixels/mm that was determined in the COPcalibration step. The processor 4102 may calculate the IPD usingtriangulation. The accuracy of the measurements of the displacementdistance d and the parallax of each view contribute to a preciseknowledge of the IPD of the stereoscopic visualization camera 300.

FIG. 46 shows a plan view of an optical schematic that is illustrativeof how the optical axis of the stereoscopic visualization camera 300 canbe measured and calibrated, according to an example embodiment of thepresent disclosure. In this example, the optical axis 4502 is misalignedfrom the mechanical axis 4504 by an angle (a), shown as 4602. The rightimage sensor 746 and the left image sensor 748 (as approximated by oneor more camera models) are aligned and focused on an object 4508 (in atarget surgical site) that is placed at a focus distance such thatparallax at the plane of the object 4508 is theoretically zero, asdepicted in the display of the left or right view of object 4508 atfocus plane 4604.

FIG. 46 also illustrated another example where the object 4508 isdisplaced along the mechanical axis the distance “d” and shown as object4508′. The displacement of the object 4508 generates parallax, whichappears in the displays of the left view 4610 as P_(L) and right view4612 as PR. In this example where the mechanical axis 4504 and theoptical axis 4502 are not coincident, the parallax magnitudes are notequal. The example processor 4102 is configured to calculate the IPD aswell as the misalignment angle α (e.g., the stereoscopic optical axis)via triangulation. The accuracy of the measurements of the displacementdistance d and the parallax of each view enable the processor 4102 toaccurately determine the IPD and the optical axis of the stereoscopicvisualization camera 300.

A similar procedure can be employed to measure, for example,misalignment of the mechanical and optical axes in the vertical plane.The combination of misalignment in, for example, the horizontal plane orvertical plane, can be combined such that a view vector can beaccurately deduced with respect to the mechanical axis. In someembodiments, the IPD and optical axis parameters may be measured atvarying levels of working distance and/or magnification. The relationsbetween IPD, optical axis, working distance, and/or magnification may bestored by the processor 4102 to the LUT 4203, another LUT, and/orcalibration registers.

Returning to FIG. 42, after the optical axis and/or IPD of the examplestereoscopic visualization camera 300 is calibrated, the exampleprocessor 4102 is configured to complete the calibration process toenable the camera 300 to be connected to the robotic arm 506 (block4212). The procedure 4200 may then end. In some embodiments, at leastportions of the example procedure 4200 are repeated if the camera 300 isreinitialized and/or if any of the calibration cannot be verified orvalidated.

It should be appreciated that the above steps of the procedure 4200 canbe performed manually or semi-manually in some embodiments. In otherembodiments, the above steps may be performed automatically andcontinuously by the processor 4200. In some embodiments, measuring canbe made through image recognition of a suitable target or any targetwith a sufficient number of objects comprising sufficient contrast toenable identification in both left and right views. In addition, theprocessor 4102 may determine or calculate parallax measurements forassessing accurate relative positions of the optical elements of thestereoscopic visualization camera 300. The processor 4102 may performoptical measurement on a real-time basis.

In some embodiments, the use of automated, iterative techniques toperform these or equivalent methods of calibration and measurement canincrease the accuracy and reduce the time and/or effort required tocalibrate and measure. For example, the working distance (and hence thedisplacement d) is accurately known by the quantity of encoder countsand the LUT 4203, as described previously. The magnification and its,for example, pixels/mm conversion factor is also accurately known by thequantity of encoder counts and the LUT 4203, as described previously.Counting of pixels in the images for disparity or object sizedetermination can be accurately performed manually or automated, forexample, as described previously using template matching. Themeasurement and storage of these values can be combined such that thestereoscopic camera model parameters and view vector can be accuratelydeduced in near real-time by the example processor 4102.

FIG. 47 illustrates a diagram of a calibrated stereoscopic visualizationcamera 300 in which the optical parameters are fully characterized. Inthe illustrated embodiment, the left and right optical axes are shownleading to imaginary left and right image sensor positions 4700, asdetermined via a camera model. FIG. 47 also shows the centralstereoscopic optical axis or view vector 4702. The imaginary left andright view components of the camera models are positioned at a focaldistance Z. In addition, the left and right view components of thecamera models are spaced apart by the measured, effective IPD. In theillustrated example, an object at the focal plane is viewed with similarstereoscopic perspective by the imaginary left and right view componentsas recorded by the image sensors 746 and 748 within the stereoscopicvisualization camera 300.

2. Calibration of the Stereoscopic Visualization Enables Fusion withAdditional Images

The example processor 4102 is configured to use the calibrationparameters/information for not only providing high-resolution clearimages, but also to align live stereoscopic images with one or moreimages/models received from the external devices 4104. The mapping ofthe calibration data related to camera model parameters in the LUT 4203and/or calibration registers enables the processor 4102 to create amathematical model of the stereoscopic visualization camera 300 that isimplemented in software, firmware, hardware and/or computer code. In anexample, the processor 4102 is configured to receive, determine, oraccess camera model parameters using, for example, the procedure 4200discussed in conjunction with FIG. 42. If a calibration has already beenperformed, the processor 4102 accesses the camera model parameters fromone or more memories 1570 and/or 4120. The processor 4102 also receives,from the device 4104, an alternative modality of the image data, such aspre-surgical images, MRI images, a 3D model of the surgical site from MMor CT data, X-ray images, and/or surgical guides/templates. Theprocessor 4102 is configured to render a synthesized stereoscopic imageof the alternative modality data using the camera model parameters. Theexample processor 4102 is also configured to provide the synthesizedstereoscopic image for display via the monitor 512. In some examples,the processor 4102 is configured to fuse the synthesized stereoscopicimage with the current stereoscopic visualization, where desirableaspects of each modality are visible and/or overlaid in identicalperspective as if acquired by a single visualization device.

In some embodiments, the parameters illustrated in FIG. 47 are used bythe processor 4102 to match a synthesized stereoscopic image ofalternative modality, for example MRI image data, to the stereoscopicperspective of the stereoscopic visualization camera 300. Thus, theexample processor 4102 uses the stored optical calibration parametersfor stereoscopic image synthesis. In an example, the processor 4102 usesthe optical calibration parameters to fuse live stereoscopic images witha three-dimensional model of a brain tumor that was imagedpre-operatively using an MRI device. The example processor 4102 uses theoptical calibration parameters to select the corresponding location,size, and or orientation of the three-dimensional model of the braintumor that matches the stereoscopic images. In other words, theprocessor 4102 selects a portion of the three-dimensional model thatcorresponds to the view recorded by the stereoscopic visualizationcamera 300. The processor 4102 may also change which portion of themodel is displayed based on detecting how the working distance,magnification, and/or orientation of the camera 300 changes.

The processor 4102 may cause a graphical representation of the model tobe overlaid of the stereoscopic images and/or cause the graphicalrepresentation of the model to appear visually fused with thestereoscopic images. The image processing performed by the processor4102 may include smoothing boundaries between the graphicalrepresentation of the model and the live stereoscopic view. The imageprocessing may also include causing at least a portion of the graphicalrepresentation of the model to have an increased transparency to enablethe underlying live stereoscopic view to also be visible to a surgeon.

In some examples, the processor 4102 is configured to generate and/orrender a depth map for every pixel in a stereoscopic image. Theprocessor 4102 may use the calibration parameters to determine, forexample, tissue depth in an image. The processor 4102 may use the depthinformation for image recognition to note tissue of interest and/oridentify instrument location to avoid inadvertent contact when thecamera 300 is mated with the robotic arm 506. The depth information maybe output by the processor 4102 to, for example, robotic suturingdevices, diagnostic equipment, procedure monitoring and recordingsystems, etc. to conduct a coordinated and at least semi-automatedsurgical procedure.

3. Calibration of the Robotic Arm Embodiment

After the stereoscopic visualization camera 300 is calibrated, asdiscussed above, it may be connected to the robotic arm 506 and/or thecoupling plate 3304. As described below, precise knowledge of theworking distance with respect to focal distance Z, provided by thestored calibration parameters, is used by the example processor 4102and/or the robotic processor 4122 for determining a position and/ororientation for the stereoscopic visualization camera 300. Thecombination of the stereoscopic visualization camera 300 with therobotic arm is configured to provide seamless transitions of variousworking distances while holding a focus or view of a target surgicalsite.

The calibration procedure for the robotic arm 506, described below, maybe executed regardless of a robotic arm type. For example, thecalibration procedure may be performed for an articulated robotic systemthat includes mechanical links connected to each other via rotaryjoints, numbering from simple one or two links and joints, to jointstructures comprising six or more joints. The calibration procedure mayalso be performed for a Cartesian robotic system that comprises a gantrywith linear joints, which uses a coordinate system with X, Y and Zdirections. A final joint of a Cartesian robotic system may comprise awrist type swiveling joint. The calibration procedure may further beperformed for a cylindrical robotic system that comprises a rotary jointat its base and one or more additional rotary and/or linear joints toform a cylindrical workspace. Moreover, the calibration procedure may beperformed for a polar robotic system that comprises an arm connected toa base via a joint that may operate in more than one rotational axis andfurther comprises one or more linear or wrist joints. The calibrationprocedure may additionally be performed for a Selective ComplianceAssembly Robot Arm (“SCARA”) system that comprises a selectivelycompliant arm operated in a primarily cylindrical fashion, which is usedfor assembly applications.

FIG. 48 illustrates an example procedure 4800 or routine for calibratingthe robotic arm 506, according to an example embodiment of the presentdisclosure. Although the procedure 4800 is described with reference tothe flow diagram illustrated in FIG. 48, it should be appreciated thatmany other methods of performing the steps associated with the procedure4800 may be used. For example, the order of many of the blocks may bechanged, certain blocks may be combined with other blocks, and many ofthe blocks described are optional. Further, the actions described inprocedure 4800 may be performed among multiple devices including, forexample the optical elements 1402, the image capture module 1404, themotor and lighting module 1406, the information processor module 1408 ofthe example stereoscopic visualization camera 300 of FIG. 14 and/orjoints R1 to R9 and robotic arm controller 4106 of FIG. 41. For example,the procedure 4800 may be performed by a program stored in the memory4120 of the robotic arm controller 4106.

In some embodiments, the coupling plate 3304 is connected to the roboticarm 506 (block 4802). If a coupling plate 3304 is not used, thestereoscopic visualization camera 300 is connected directly to theconnection or coupling interface 3450 of the robotic arm 506. If thecoupling plate 3304 is used, the stereoscopic visualization camera 300is connected to the coupling plate (block 4804). As discussed above,first end 3702 of the coupling plate 3304 is connected to the roboticarm 506 and the second end 3704 of the coupling plate 3304 is connectedto the stereoscopic visualization camera 300.

After the example stereoscopic visualization camera 300 is connected tothe robotic arm 506, the example processor 4102 and/or the robotic armcontroller 4106 are configured to calibrate the camera and its viewvector into a coordinate system originated around the stationary base3404 of the robotic arm 506 (block 4806). The coordinate system isreferred to herein as “robot space” or “robotic space”. During thiscalibration step, known movements to the robotic arm 506 are used by theprocessor 4102 and/or the robotic arm controller 4106 to determine anorientation and a location of a view vector and object plane of thecamera 300 during visualization of a target surgical site.

In some embodiments, the mechanical features of the camera 300, thecoupling plate 3304, and the robotic arm 506 exist such that, whenmechanically connected together, the relationship between the camera300, the coupling plate 3304, and the robotic arm 506 is uniquelydetermined and known. In these embodiments, the processor 4102 and/orthe robotic arm controller 4106 determine the position, direction,and/or orientation of the view vector from the known mechanical geometryof the camera 300, the coupling plate 3304, and the robotic arm 506.

In other embodiments where the mechanical features do not exist, theexample processor 4102 and/or the robotic arm controller 4106 areconfigured to perform a routine to accurately determine a spatialrelationship between the camera 300 and the robotic arm 506 in robotspace. The processor 4102 and/or the robotic arm controller 4106 movethe stereoscopic visualization camera 300 to a start position, which mayinclude a stow position, a re-orientation position, or a surgicalposition. The stereoscopic visualization camera 300 then moves thecamera from the start position to a position that approximatelyvisualizes a calibration target located on the stationary base 3404 ofthe robotic arm 506. The calibration target may be located, for example,at a convenient area of the cart 510 in a position within the motionsphere of the robotic arm 506. Some examples of the calibration targetinclude, for example, small spheres or other uniquely recognizableobjects that can be located relative to each other (in two-dimensionalor stereoscopic images) in a unique, known orientation. The coordinatesof the spheres are fixed and known with respect to the cart 510 andstationary base 3404, and are hence known in robot space. The processor4102 and/or the robotic arm controller 4106 are configured to store thecoordinates to, for example, the memory 4120.

During the calibration, the processor 4102 and/or the robotic armcontroller 4106 receive view vector data 4807 regarding workingdistance, magnification, stereoscopic optical axis, and/or IPD. Thestereoscopic visualization camera 300 is set to visualize the spheres atthe calibration target simultaneously and determine their positionthrough the use of parallax in the stereoscopic image. The processor4102 and/or the robotic arm controller 4106 records the positions of thespheres in an initial coordinate system, for example, X, Y, and Z withrespect to a fiducial in the camera 300 (i.e. “camera space). The X,Y,Zposition may correspond to an origin location, and be defined in a fileor LUT as being the origin or having other known coordinate values. Theprocessor 4102 and/or the robotic arm controller 4106 also use outputdata from joint sensors to determine position and orientation of thejoints and links in the robotic arm 506. The processor 4102 and/or therobotic arm controller 4106 also receive position information todetermine a position and orientation of the coupling device 3304.Together, the position and orientation of the robotic arm 506 and thecoupling device 3304 enable the processor 4102 and/or the robotic armcontroller 4106 to determine a pose of the camera 300. The processor4102 and/or the robotic arm controller 4106 are configured to perform acoordinate transformation between the camera space and robot space basedon the positions of the spheres of the calibration target as recorded bythe camera, and as the positions of the robotic arm 506 and/or couplingplate 3304. The processor 4102 and/or the robotic arm controller 4106may store the coordinate transformation to the LUT 4203, a different LUTfor the robotic arm 506, and/or one or more calibration registers.

In some embodiments, the camera 300 is moved to record images ofmultiple calibration targets located either on a cart 510, a ceiling, awall, and/or within a surgical area. Each of the calibration targets mayhave a unique orientation that enables it physical X,Y,Z location to beidentified. The processor 4102 and/or the robotic arm controller 4106perform additional coordinate transformations for each of thecalibration targets and store the transformations to one or more LUTsand/or registers.

In other embodiments, the processor 4102 and/or the robotic armcontroller 4106 may use alternative methods to calibrate the camera 300to robot space. In this context, “calibration” is taken to mean“registration”, where the processor 4102 and/or the robotic armcontroller 4106 are configured to calculate registration over a widespace in which the registration may vary. For example, a system can beused where a separate stereoscopic camera is used to observe and locatecalibration targets on the cart 510 as well as similar calibrationtargets which are installed on the camera 300 and/or on a patient orsurgical bed. The processor 4102 and/or the robotic arm controller 4106are configured to model and track the camera 300, which is modeled andtracked as a surgical instrument with a view vector and workingdistance. The view vector and working distance define parameters foraccurately visualizing a target surgical site. In these otherembodiments, the other camera determines and reports location andorientation information for the coordinate frame of each such instrumentin a reference frame, such as the stereoscopic camera 300. Then, usinglinear algebra, the poses and/or locations of instruments relative toeach other are calculated by the processor 4102 and/or the robotic armcontroller 4106, thereby resulting in a calibration of the camera 300 tothe robot space.

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 are also configured to calibrate for the coupling plate3304. In some instances, the coupling plate 3304 includes one or moreswitches that activate depending on a position of joints R7 to R9. Theknown position of the switches is used by the processor 4102 and/or therobotic arm controller 4106 as part of the coordinate transformation.Additionally or alternatively, the coupling plate 3304 is calibrated bycausing the robotic arm 506 to move while images from the camera 300 aremonitored to determine orientation. In an example where the couplingplate 3304 is orientated as shown in FIG. 37, the robotic arm 506 iscommanded to move in a direction relative to an assumed orientation (forexample, moving the camera 300 along the z-axis). If the assumedorientation is as shown in FIG. 37, wherein the camera 300 is aimeddownward, a downward movement of the robotic arm 506 should cause anobject in the image to get larger as the camera 300 gets closer. If, forexample, the object in the image, instead moves sideways or up/down,then the processor 4102 and/or the robotic arm controller 4106 areconfigured to detect the motion and determine that the assumedorientation is incorrect. The processor 4102 and/or the robotic armcontroller 4106 may generate an error and prompt an operator for thecorrect orientation and/or determine the correct orientation based onthe detected movement in the images. The change in the image frommovement of the camera 300 is deciphered automatically through use of,for example, image matching template algorithms, as describedpreviously. In some embodiments, the use of matching template algorithmsby the processor 4102 and/or the robotic arm controller 4106 determinesjoint orientation at the coupling plate 3304, which is stored to a LUTfor calibration.

FIG. 49 shows a diagram that is illustrative of how the stereoscopicvisualization camera 300 and/or the robotic arm 506 are calibrated torobot space, according to an example embodiment of the presentdisclosure. In the illustrated embodiment, each of joints R1 to R9 andcorresponding links are modeled based on rotational capabilities and/orlengths. The memory 4120 may store the mathematical parametersassociated with the model. Further, the processor 4102 and/or therobotic arm controller 4106 may use the mathematical model to determine,for example, a current position of the robotic arm 506 and/or camera300, which may be used for calculating how joints are to be rotatedbased on intended movement provided by an operator.

In the illustrated example, joint R1 is provided at a coordinateposition of (0,0,0). The lengths between the joints R1 to R9 correspondto a length of the links. In the illustrated example, the stereoscopicvisualization camera 300 is modeled as a robot end effector that isconnected to nine couplers. The three-dimensional space shown in FIG. 49is modeled using a sequence of ten homogeneous transformations, whichmay include matrix multiplications. The first six frames or joints R1 toR6 represent the forward kinematics of the robotic arm 506, and may becalculated using the Denavit-Hartenberg parameters of a robotic arm. Thenext three frames or joints R7 to R9 represent the transform from thetool-tip of the robotic arm 506 to a tip of the coupling plate 3304. Thelast frame R10 represents the transform from the tool-tip of thecoupling plate 3304 to the control point of the stereoscopicvisualization camera 300.

Frame or joint R7 represents the pitch joint of the coupling plate 3304,which, can change between 0° and 90°. Frame or joint R8 represents theyaw joint of the coupling plate 3304, and can change between −90°, 0°,and 90°, depending on the yaw configuration. Joints R7 to R9 of thecoupling plate may include a voltage source and a potentiometer. Theconnector 3450 and/or the coupler controller 4130 of the robotic arm 506may include an I/O tool-tip connector that is configured to receive avoltage output from the potentiometer. The processor 4102 and/or therobotic arm controller 4106 are configured to receive the output voltageand correspondingly determine pitch and yaw angles of the coupling plate3304. The processor 4102 and/or the robotic arm controller 4106 combinesthe pitch and yaw information of the coupling plate with sensor outputdata from joints R1 to R6 of the robotic arm 506 to calculate positionof the frames R1 to R10 to determine the three-dimensional position ofthe robotic arm 506, the coupling plate 3304, and/or the camera 300.

The control point represents frame 10 at the very end of the kinematicchain, and is fully programmable in terms of position based on whichfeature is selected. For example, if an operator selects an assisteddrive feature, the processor 4102 and/or the robotic arm controller 4106are configured to set the control point representative of the camera 300to be inside of the camera along an axis of rotation of the control arms304. In another example, if an operator selects a lock-to-targetfeature, the processor 4102 and/or the robotic arm controller 4106 areconfigured to set the control point of the camera 300 to an origin of anoptical axis view vector.

Returning to FIG. 48, the after calibrating the camera 300 to robotspace, the processor 4102 and/or the robotic arm controller 4106 areconfigured to calibrate the robot space to patient space (block 4808).Calibration of patient space is need to enable the stereoscopicvisualization platform 516 to make accurate visualizations of a patient,where the orientation between robot system and patient is needed. Insome embodiments this orientation is fixed. In other embodiments theorientation, if varying, is sensed and known. In some embodiments apatient is placed in an operating room bed and registered to the bedusing one or more fiducials 4809. For example, if a patient isundergoing brain surgery, they are secured to a bed and an externalframe is fixed to their skull. The frame is observable by thestereoscopic visualization camera 300 and may comprise fiducials 4809 inan arrangement such as that of the calibration target where two or morenon-collinear objects of known locations are visible simultaneously,such that the position and orientation of the frame, and hence thepatient's skull, is capable of being determined. Other embodiments mayuse fiducials 4809 that are implanted into a patient and are visible inMM or similar images. Such fiducials 4809 can be used to accuratelytrack and register a patient's skull as well as the MM image to acoordinate system representative of patient space. Further, otherembodiments may use image recognition of features native to the patientthemselves. For example, facial or similar recognition using biometricdata, in-situ x-ray, or similar alternative modality imaging can be usedto precisely locate a position and orientation of the patient. Inanother example, a model of a surface of a patient's face can bedetermined using one or more depth map calculations as described above,and surface matching functions performed by the processor 4102 and/orthe robotic arm controller 4106.

In an embodiment, a position and orientation of an operating room bedwith respect to robot space is fixed and determined. Some embodimentscomprise a rigid frame which mechanically registers the bed to, forexample, fittings on the cart 510 in a known position and orientation.Alternatively, the bed can be fixed with respect to the robotic arm 506and fiducials can be used to determine position and orientation. Forexample, the robotic cart 510 and bed can be anchored to the floor andfixed for the duration of the procedure.

After visualization of the patient's fiducials 4809 by the camera 300,their position and orientation in robot space can be deciphered andstored by the processor 4102 and/or the robotic arm controller 4106,where coordinate system transformations from robot space to patientspace are enabled. It is noted that coordinate system transformationsfrom one space to another are generally selectable and reversible. Forexample, it may be more efficient to transform desired camera motions orposes into robot space to enable the processor 4102 and/or the roboticarm controller 4106 to determine discrete joint motion and orientation.Alternatively, it may be easier and more efficient to presentinformation to a surgeon on the display monitor 512 in patient space.Location of points and vectors can be transformed by the processor 4102and/or the robotic arm controller 4106 to be respective of most anycoordinate system, for example, a cart origin, a patient referenceframe, GPS, and/or other coordinate systems as desired.

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 are configured to use automated, iterative techniques toperform these or equivalent methods of robot/patient space calibrationand measurement to increase accuracy and reduce calibration time. Inexemplary embodiments, the displacement and orientation of thestereoscopic visualization camera 300 with respect to fiducials isaccurately known by the processor 4102 and/or the robotic arm controller4106. Motion of the robotic arm 506 can be accurately performed, and thesubsequent images of fiducials can be accurately analyzed. Thevisualization and knowledge of the calibration parameters can becombined by the processor 4102 and/or the robotic arm controller 4106such that measurement, and hence calibration can be accurately performedin an automated manner. This is important, for example, to maintainaccurate calibrations from one surgical procedure and one patient to thenext.

In some examples, the processor 4102 and/or the robotic arm controller4106 are configured to determine boundaries of the robotic arm 506and/or camera 300 relative to the patient space and/or robot space. Theboundaries represent virtual limits that are implemented in the softwareto prevent the robotic arm 506 and/or the camera 300 from contacting orescaping defined areas or spaces. In some examples, the limits aredefined in one or more LUTs or registers stored in the memory 4120 asscale factors that are applied to joint movement signals by theprocessor 4102 and/or the robotic arm controller 4106. The magnitude ofthe scale factor is decreased to zero as the limit to each individualboundary is approached. For example, the joint rotation amount and speedmay be determined based on operator input. However, the processor 4102and/or the robotic arm controller 4106 scales the joint rotation speedby the scale factor before sending the signal(s) to the appropriatejoint(s). In addition, the processor 4102 and/or the robotic armcontroller 4106 may maintain the rotation amount such that the jointmoves the desired amount, but at a reduced speed, until the jointreaches the boundary. It should be appreciated that a joint in arotation area where a scale factor is applied may not have a scalefactor applied if the desired movement is away from the boundary. Thus,the processor 4102 and/or the robotic arm controller 4106 may apply ascale factor to certain joints while applying a scale factor of ‘1’ toother joints based on a current position and estimated desired movementfrom an operator.

The scale factors are strictly between zero and one, which enableschaining them together and enables the software to support an infinitenumber of possible boundaries. The scale factors may be lineallydecreased as a boundary is approached, which causes a gradually slowingof the rotation of joints R1 to R9 as the robotic arm 506 approaches aboundary. In other examples, the scale factors may decreaseexponentially as a boundary is approached.

Generally, operators typically focus their attention on the surgicalfield or the stereoscopic image on the display monitor 512. As such, theoperators are typically unaware of the position of the individual linksof the robotic arm 506 and/or the coupling plate 3304. Therefore, it isnot always intuitive when the robot arm 506 is about to reach a limit orimpact another part of the robot arm 506. The joint limits may thereforealways be active and prevent any part of the robot arm 506 from hittingitself or putting the joints in a singular configuration, such as elbowlock. The example processor 4102 and/or the robotic arm controller 4106are configured to determine the scale factor based on a current positionof the robotic arm 506. The processor 4102 and/or the robotic armcontroller 4106 may also take into account intended movementinstructions provided by an operator to determine which scale factor isto be applied. Based on current and/or anticipated movement, theprocessor 4102 and/or the robotic arm controller 4106 calculates thescale factors based on distances in joint angle space using, forexample, one or more LUTs. The joint angle spacing may define certaincombinations of joint angles that are known to cause joint lock or causethe robotic arm 506 to hit itself. As such, the joint angle spacingdetermination is based on determining and comparing current (and/oranticipated) movements of joints relative to each other.

In addition to the boundaries for the robotic arm 506, the memory 4120may store boundaries that relate to Cartesian limits that prevent therobotic arm 506 from hitting the cart 510, the robotic arm from hittingthe display monitor 512, and/or the camera 300 from hitting the roboticarm 506. The processor 4102 and/or the robotic arm controller 4106 mayuse, for example, the coordinate system discussed in conjunction withFIG. 49 for determining and/or applying the Cartesian limits. In someexamples, the limits may be relative or anchored to a certain link. Assuch, when the link is moved in the 3D space, the boundary around itmoves accordingly. In other examples, the limits are static and fixed tocertain coordinate planes or lines within the 3D space shown in FIG. 49.The processor 4102 and/or the robotic arm controller 4106 may apply thelimits by calculating or determining scale factors in Cartesian spaceand applying the forward kinematic transform.

The example processor 4102 and/or the robotic arm controller 4106 mayalso determine a patient boundary, which defines a virtual place that nopoint of the robotic arm 506 and/or camera 300 can violate. Patientboundaries may be determined by calculating scale factors in Cartesianspace for a distance of each positional joint on the robotic arm 506and/or the coupling plate 3304 to a location of a boundary plane. Theboundary plane, as shown in orientation 5002 of FIG. 50 is implementedas an X,Y plane located at some vertical Z location for non-pitchedconfigurations. For pitched configurations, such as patient semi-sittingshown in orientation 5004 of FIG. 50, the boundary plane is set as a Y,Zplane located at either positive or negative X values depending on thedirection the camera 300 faces.

The example boundaries discussed above may be stored to the memory 4120as default boundaries and/or determined by the processor 4102 and/or therobotic arm controller 4106 prior to a surgical procedure. In someembodiments, certain boundaries may be accessed or determined based onan inputted type of surgical procedure to be performed. For example,patient boundaries may be determined by the camera 300 imaging thepatient and determining patient depth using calibrationinformation/parameters. The processor 4102 and/or the robotic armcontroller 4106 may then create and apply a boundary to a specifiedlocation above or next to the patient. Similar boundaries may be createdafter detection of monitors, surgical staff, or surgical instruments.

For instance, boundaries can be determined around the use of a specificsurgical tool such that tools of larger size or tools that pose certainrisks if contacted. The example processor 4102 and/or the robotic armcontroller 4106 may receive an input of the tool type and/or detect thetool in the stereoscopic images using image analysis. In other examples,the processor 4102 and/or the robotic arm controller 4106 calculatedepth information in relation to a surgical instrument to determine itssize, orientation, and/or position. The example processor 4102 and/orthe robotic arm controller 4106 translate the image of the surgicalinstrument into the coordinate system, such as the one discussed inconnection with FIG. 49. The processor 4102 and/or the robotic armcontroller 4106 also apply scale factors having a value less than ‘1’ toareas that correspond to a location of the surgical instrument, therebypreventing the robotic arm 506 and/or the camera 300 from inadvertentlycontacting the surgical tool. In some instances, the processor 4102and/or the robotic arm controller 4106 may track a movement of thesurgical tool during a procedure and change the boundary accordingly.

FIG. 51 illustrates an example of how the rotational joint speed of therobotic arm 506 and/or the coupling plate 3304 is scaled based ondistance to a boundary, according to an example embodiment of thepresent disclosure. Graph 5102 shows a velocity of rotation for joint R1and graph 5104 shows a shoulder angle (e.g., rotation position) 5110 inrelation to a first zone 5112 that corresponds to an area close to aboundary where a scale factor is reduced from a value of ‘1’ and asecond zone 5114 that corresponds to the boundary where the scale factoris reduced to a value of ‘0’.

FIG. 51 shows that as the robotic arm 506, and in particular joint R1causes the at least one link and/or the stereoscopic visualizationcamera 300 to approach the first zone 5112, the rotational velocity isdynamically scaled with respect to the distance to the second zone 5114.Then, when the at least one link and/or the stereoscopic visualizationcamera 300 reach the second zone 5114, the scale factor is reduced to avalue of ‘0’ and all rotational joint movement toward the boundary isstopped. In other words, as the robotic arm 506 and the stereoscopicvisualization camera 300 approach a limit or boundary, the processor4102 and/or the robotic arm controller 4106 causes a rotational speed ofat least some of joints R1 to R9 to decrease and eventually reach avelocity of ‘0’ degrees/second when the second zone 5114 is reached (asshown between 20 and 30 seconds in the graphs 5102 and 5104). The graphalso shows that when the at least one link and/or the stereoscopicvisualization camera 300 are moved away from the second zone 5114, theprocessor 4102 and/or the robotic arm controller 4106 use a scale factorvalue of ‘1’ since the second zone 5114 is not being approached.

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 are configured to cause the display monitor 512 or otheruser interface to display one or more graphical icons representative ofa status of the robotic arm 506. For example, a green icon may bedisplayed when the robotic arm 506 and/or camera 300 are located in azone or area where scale factors have a value of ‘1’. Additionally, ayellow icon may be displayed when the robotic arm 506 and/or camera 300are located within the first zone 5112 to indicate joint rotationalspeed is slowed. Further, a red icon may be displayed when the roboticarm 506 reaches the second zone 5114 or a boundary/limit to indicatethat no further movement beyond the boundary is possible.

Returning to FIG. 48, after the robot space boundaries are determined,the example processor 4102 and/or the robotic arm controller 4106 areconfigured to enable the robotic arm 506 for operation with thestereoscopic visualization camera 300 (block 4812). This may includeenabling the robotic arm 506 and the stereoscopic visualization camera300 to be used during a surgical procedure. This may also includeenabling features, such as assisted drive and/or lock-to-target.Additionally or alternatively, this may include enabling one or moreuser controls at one or more of the input devices 1410 of FIG. 41. Theexample procedure 4800 ends after the robotic arm 506 is enabled withthe stereoscopic visualization camera 300. The example procedure 4800may repeat if the stereoscopic visualization platform 516 isreinitialized, experiences a detected failure, or the calibration cannotbe validated.

D. Stereoscopic Visualization Camera and Robotic Arm OperationEmbodiments

The example stereoscopic visualization camera 300 is configured tooperate in conjunction with the robotic arm 506 and/or the couplingplate 3304 to provide enhanced visualization features. As discussedbelow in more detail, the enhanced features include an extended focus,automated focal tip positioning, providing a measurement of distancesbetween objects in an image, providing robotic motion with conjoinedvisualization, sag compensation, image fusion, and storage ofvisualization positions/orientations. The enhanced visualizationfeatures may also include assisted-drive capability of the robotic arm506 and a lock-to-target capability that enables the camera to be lockedonto a specific view while enabling an orientation of the robotic arm506 and/or the coupling plate 3304 to be changed.

1. Extended Focus Embodiment

In some embodiments, the robotic arm 506 and/or the coupling plate 3304may provide an extended focus of the camera 300. As discussed above inconnection with FIG. 43, the stereoscopic visualization camera 300includes the main objective assembly 702 for changing a workingdistance. To focus on an object in the surgical site, the main objectiveassembly 702 changes a focus distance from just before the object tojust past the object. However, in some instances, the main objectiveassembly 702 reaches a mechanical limit of the front working distancelens 408 before the best focus is achieved.

The example processor 4102 and/or the robotic arm controller 4106 areconfigured to detect when a mechanical limit is reached and/or determinethat a mechanical limit is about to be reached for the lens 408 andaccordingly adjust a position of the robotic arm 506 instead. Theprocessor 4102 and/or the robotic arm controller 4106 is configured touse the robotic arm 506 to extend focus by computing a view vector ofthe camera 300 and causing the robotic arm 506 to be actuated along theoptical axis. The processor 4102 and/or the robotic arm controller 4106determine a distance needed to achieve focus using the above-describedcalibration parameters of the stereoscopic visualization camera 300. Forexample, as discussed above, a position of the front working distancelens 408 is mapped to a physical working distance of the main objectiveassembly 702 to a target object. The distance provides an estimate as tohow far a center of the camera 300 is from the target object.Additionally, the calibration parameters may include a mapping betweenmotor or encoder steps for the front working distance lens 408 toworking distance to provide an estimation of distance needed to achievea certain working distance or focus. Accordingly, the processor 4102and/or the robotic arm controller 4106 may read a current encoder valueof the front working distance lens 408 and determine a number in metersthat represents a vertical distance from the camera 300 to the targetobject. In other words, the processor 4102 and/or the robotic armcontroller 4106 convert the lens movement (in encoder counts) into aphysical distance in the robot space. The processor 4102 and/or therobotic arm controller 4106 then determine joint rotational speeds,directions, and/or durations (e.g., a movement sequence) to that willcause the robotic arm 506 to move the determined distance along theoptical axis. The processor 4102 and/or the robotic arm controller 4106then transmits one or more signals to the appropriate jointscorresponding to the movement sequence to cause the robotic arm 506 toprovide an extended focus. In some instances, the processor 4102 and/orthe robotic arm controller 4106 may apply a scale factor before thesignals are transmitted to joints R1 to R9 of the robotic arm 506 and/orthe coupling plate 3304.

It should be appreciated that the extension of focus causes an automatedmovement of the robotic arm 506. In other words, the robotic arm 506 cancontinue motion of the camera 300 through the point of best focus. Inaddition, the movement of the robotic arm 506 occurs without inputs froman operator to move the robotic arm, but rather, operator imagesregarding the changing of a focus. In some instances, the processor 4102and/or the robotic arm controller 4106 may adjust the focusautomatically to maintain a clear image.

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 are configured to move the robotic arm 506 along thecamera's working distance in response to a single button press via theinput device 1410. This feature enables an operator to fix a motorposition of the main objective assembly 702 and obtain focus by movingthe robotic arm 506 and/or the coupling plate 3304. This “robot autofocus” feature or procedure is accomplished by the processor 4102 and/orthe robotic arm controller 4106 estimating or determining a distancefrom a front of the main objective assembly 702 to a target, asdiscussed above in connection with FIG. 43. The processor 4102 and/orthe robotic arm controller 4106 is configured to use the determineddistance with a feedback law to command a vertical velocity of therobotic arm 506 (or velocity along an optical axis of the camera 300)until the determined distance reaches a value of ‘0’. The processor 4102and/or the robotic arm controller 4106 may use this autofocus algorithmanytime during a procedure to bring a target object into focus. In someembodiments, the processor 4102 and/or the robotic arm controller 4106may use movement of the robotic arm 506 and/or the coupling plate 3304from the last time autofocus was used as a seed or starting point whensearching for a direction of autofocus, thereby improving the speed andaccuracy of getting a target object into focus.

It should be appreciated that the example processor 4102 and/or therobotic arm controller 4106 may be configured to cause the robotic arm506 and/or the coupling plate 3304 to move in addition to oralternatively from moving the front lens set 714, the lens barrel set718, and/or the final optical set 742, each of which may be movable by arespective motor that has encoder counts mapped to position, focus,working distance, and/or magnification. For example, the processor 4102and/or the robotic arm controller 4106 may cause the robotic arm 506 tomove along an optical axis when any of the front lens set 714, the lensbarrel set 718, and/or the final optical set 742 is about to approach amovement limit. In some examples, the processor 4102 and/or the roboticarm controller 4106 may cause the robotic arm 506 to move first to aposition that is roughly in focus or near-focus, and then adjust thefront lens set 714, the lens barrel set 718, and/or the final opticalset 742 to bring the target image into near-ideal focus.

2. Automated Focal Tip Positioning Embodiment

In some embodiments, the robotic arm 506 and/or the coupling plate 3304may be operated in conjunction with the stereoscopic visualizationcamera 300 to provide automated focal tip positioning. In theseembodiments, the processor 4102 and/or the robotic arm controller 4106is configured to position the camera 300 for visualization of a targetsurgical site without information or feedback of a specific image andits contents. The processor 4102 and/or the robotic arm controller 4106may use the calibrated camera model parameters, discussed above inconnection with FIGS. 42 and 49 to perform open loop camera positioning.The processor 4102 and/or the robotic arm controller 4106 may cause therobotic arm 506 to position the stereoscopic visualization camera 300such that a focal point or tip of the camera is in a scene. Thestereoscopic visualization camera 300 determines an aiming direction forthe camera 300, with respect to a coordinate system, based oncalibration information regarding a pose of the robotic arm 506 and/orthe coupling plate and optical calibration parameters of the camera 300.The processor 4102 and/or the robotic arm controller 4106 maycharacterize the aiming by a geometrically-defined view vector, which isaligned coincidentally with the stereoscopic optical axis of the camera300, with respect to the coordinate system of the robotic arm 506.

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 are configured to execute an initialization routine toalign calibration parameters and/or other memory data to an actualphysical reference position, which may be used for tip positioning. Forexample, the processor 4102 and/or the robotic arm controller 4106 maycause the robotic arm 506 and/or the coupling plate to move to a hardstop at “position 0”, where all the position data fields are set to 0(or 0,0,0 in a three-dimensional space). Further motions are maderelative to this point and the position data is updated according to,for example, encoder counts of the various joint motors of the roboticarm 506 and/or coupling plate 3304.

In other embodiments, the processor 4102 and/or the robotic armcontroller 4106 may determine or set a tip position of the camera 300based on one or more visualization parameters. For example, theprocessor 4102 and/or the robotic arm controller 4106 may use acenter-of-projection location as a proximal end of a view vector (e.g.,a “starting point” for aiming the camera 300). In some surgical systems,this point on a surgical instrument is referred to as the “hind” point,and may be provided in relation to the tip. The processor 4102 and/orthe robotic arm controller 4106 calculate a view vector direction fromthe tip and hind points to determine an aim of the camera 300 withrespect to the coordinate system of the robotic arm 506.

Additionally or alternatively, the processor 4102 and/or the robotic armcontroller 4106 may determine a focus distance for calculating a rangeof a focus plane of a stereoscopic image from the center-of-projection.The center of the image at the focus plane is the “tip” point. Theprocessor 4102 and/or the robotic arm controller 4106 may use acalibrated working distance to determine the actual, spatial, physicaldistance from the camera 300 to the tip point. Further, the processor4102 and/or the robotic arm controller 4106 may determine themagnification, as discussed above in regards to magnificationcalibration.

3. Distance Measurement Embodiment

In some embodiments, the robotic arm 506 and/or the coupling plate 3304may be operated in conjunction with the stereoscopic visualizationcamera 300 to provide distance measurements and/or depth measurementsbetween objects in a stereoscopic image. For example, the processor 4102may determine dimensionally a center of a focal point or tip of thecamera 300 with respect the coordinate system of the robotic arm 506using optical calibration parameters transformed to robot space. Asdiscussed above in connection with FIGS. 45 and 46, a view vector andleft/right parallax information of any point in an image can be used bythe processor 4102 to calculate its position in three-dimensions throughtriangulation with respect to the tip, or to any other point in theimage. This triangulation enables the processor 4102 to map any point inan image to the robotic coordinate system. As such, the processor 4102can calculate locations and/or depths of multiple objects and/orlocations of different portions of an object with respect to the samecoordinate space of the robotic arm 506, which enables a distancemeasurement and/or depth measurement to be determined between theobjects.

The processor 4102 may cause the distance and/or depth measurementinformation to be displayed visually over and/or in conjunction with thestereoscopic image. In some instances, an operator may use the inputdevice 1410 to select two or more objects by selecting the objects on ascreen or pointing directly to the actual objects in the patient using afinger or surgical instrument. The processor 4102 receives theindication of the selection and accordingly determines the coordinatesof the objects and the distances therebetween. The processor 4102 maythen display a ruler graphic and/or values indicative of the distances(and/or an indication of the selected objects) in conjunction with thestereoscopic images.

Further, the tracking of objects enables locations of other objects thatwere previously imaged (or are provided in other images) to be storedand later compared. For instance, the camera 300 may move to a locationwhere at least some of the objects are outside of the current FOV.However, an operator can instruct the processor 4102 to determine adistance between an object within the FOV and a previously imaged objectthat is currently outside the FOV.

In some embodiments, the processor 4102 may use the coordinates ofobjects for fusing digital images or models from alternate modalityvisualizations, such as Mill images, X-ray images, surgical templates orguidelines, pre-operative images, etc. The example processor 4102 isconfigured to use object locations in the coordinate plane as well asdepth information to properly scale, orientate, and position thealternate modality visualization. The processor 4102 may select at leasta portion of the alternate modality visualization that has identicalfeatures (e.g., objects) in a displayed stereoscopic image. Forinstance, the processor 4102 may use an image analysis routine tolocate, in a stereoscopic image, a blood vessel pattern, a scar, adeformity, or other viewable physical structure or object. The processor4102 then locates the identical features in the alternate modalityvisualization. The processor 4102 selects a portion of the alternatemodality visualization that includes the identical features. Theprocessor 4102 may then use coordinates, depths, and/or distancesbetween the features in the stereoscopic image for scaling, rotating,and/or orientating the selected portion of the alternate modalityvisualization. The processor may then fuse the adjusted portion of thealternate modality visualization with the stereoscopic image(s). Theprocessor 4102 may track how the identifiable objects move relative toeach other and/or relative to the FOV to determine how the fused imageis to be accordingly updated. For example, movement of the camera 300 toanother surgical location may cause the processor 4102 to select anotherportion of the pre-surgical image for fusion with the stereoscopicimages of the other surgical location.

In some instances, the processor 4102 and/or the robotic arm controller4106 may cause the robotic arm 506 to move to track a movement of anobject in the FOV. The processor 4102 uses the coordinate position ofthe object to detect movement or obfuscation. In response to thedetected movement or obfuscation, the processor 4102 and/or the roboticarm controller 4106 are configured to determine how the robotic arm 506and/or the coupling plate 3304 are to be moved to track the movement ofthe object or overcome the obfuscation. For example, the processor 4102and/or the robotic arm controller 4106 may move the robotic arm 506 in acircular path to visualize a point on a patient's retina from multipledirections to avoid reflections or obfuscation from tools.

4. Image Fusion Embodiments

As discussed above, the processor 4102 is configured to fuse an imagefrom an alternate modality to live stereoscopic images. For example, ifa surgeon is operating on a patient with a deep brain tumor, the surgeoncan instruct that the processor 4102 visualize an MRI image of the braintumor in the proper location and at the proper depth and stereoscopicperspective as their live image from the camera 300 on the displaymonitor 512. In some embodiments, the processor 4102 is configured touse distance and/or depth measurement information of one or more objectsin the FOV for fusing with the alternate modality view. The processor4102 may also provide for imaging fusion using the stereoscopic opticalaxis (e.g., view vector), the IPD, and/or the camera model parametersthat were calculated in the calibration steps discussed in connectionwith FIG. 42 and stored to one or more LUTs. The use of the opticalcalibration parameters enables the processor 4102 to display analternate modality image as if the image was acquired by thestereoscopic visualization camera 300. The processor 4102 may use theoptical calibration parameters of the camera to model, scale, or modifyalternate modality images based on an effective IPD of the camera 300such that the alternate modality image is viewed at a distance Z from afocus point in the surgical site, given the applied working distance andmagnification of the camera 300.

FIG. 52 shows a diagram of an example procedure 5200 for fusing an imagefrom an alternate modality visualization with stereoscopic image(s),according to an example embodiment of the present disclosure. Althoughthe procedure 5200 is described with reference to the flow diagramillustrated in FIG. 52, it should be appreciated that many other methodsof performing the steps associated with the procedure 5200 may be used.For example, the order of many of the blocks may be changed, certainblocks may be combined with other blocks, and many of the blocksdescribed are optional. Further, the actions described in procedure 5200may be performed among multiple devices including, for example theoptical elements 1402, the image capture module 1404, the motor andlighting module 1406, the information processor module 1408 of theexample stereoscopic visualization camera 300 of FIG. 14 and/or jointsR1 to R9 and robotic arm controller 4106 of FIG. 41. For example, theprocedure 5200 may be performed by a program stored in the memory 1570of the processor 4102.

The example processor 4102 of procedure 5200 is configured to useoptical calibration parameters to render, for example, previouslygenerated three-dimensional MM data of a patient as a stereoscopic imagewith proper perspectives as a stereoscopic image recorded by the camera300. The processor 4102 may receive, for example, an alternate modalityvisualization, such as the MM data, from device 4104 of FIG. 41 (block5202). The processor 5202 may also receive an input 5203 via an inputdevice 1410 indicative that the alternate modality visualization is tobe fused with stereoscopic images recorded by the stereoscopicvisualization camera 300 (block 5204).

During the procedure 5200, when a surgeon positions the camera 300 at adesired orientation and position for a surgical procedure, pose data5205 is obtained by the processor 4102 (block 5206). The pose data 5201may include positions of the robotic arm 506, the coupling plate 3304,and/or the stereoscopic visualization camera 300. The processor 4102also accesses magnification and working distance optical calibrationparameters 5207 related to the camera 300 from one or more LUTs, such asthe LUTs 4203 of FIG. 42 (block 5208). The processor 4102 uses the posedata 5205 in conjunction with the magnification and working distanceoptical calibration parameters 5207 to determine a stereoscopic axis andIPD for the camera 300 (block 5210). The processor 4102 applies the posedata, stereoscopic axis data, IPD data, and/or the optical calibrationparameters to select at least a portion of the MM data and/or modify,scale, orientate, partition, etc. the selected portion of the Mill datasuch that the selected portion is provided at a perspective of a view ofthe patient's brain as viewed by the stereoscopic visualization camera300 (block 5212). The processor 4102 is configured to apply thestereoscopic optical axis view vector and IPD for rendering the selectedportion of Mill data into a stereoscopic image corresponding to thecurrent live view of the camera 300 (block 5114). The processor 4102 maythen fuse the stereoscopic Mill image with live stereoscopic image(s)from the camera 300, as discussed herein (block 5216).

As discussed above, the processor 4102 may use an object or feature forpositioning or fusing the rendered Mill data with the stereoscopicimage(s) from the stereoscopic visualization camera 300. For example,the processor 4102 may use one or more image analysis routines foridentifying distinct features or objects in a stereoscopic image,locating the same distinct features in the rendered stereoscopic MMdata, and laying the rendered stereoscopic Mill data over theappropriate portion of the camera stereoscopic image(s) such that thefeatures or objects are aligned and have the same scale, size, depth,orientation, etc. The processor 4102 may make the rendered stereoscopicMM data at least partially transparent to enable the live image(s) toalso be viewable. Additionally or alternatively, the processor 4102 mayadjust a shading at a border of the rendered stereoscopic Mill data toreduce visual contrasts between the rendered stereoscopic Mill data andthe camera stereoscopic image(s). The example procedure 5200 of FIG. 52may then end.

The example procedure 5200 enables the brain tumor to be visualized bythe surgeon in an accurate location relative to the stereoscopic imagesof the camera 300. The surgeon may use this fusion visualizationespecially partway through a surgical procedure. For example, thesurgeon can see the as yet unexposed tumor in a manner best described as“x-ray vision” below a current level of dissection. Control of thetransparency of live or rendered stereoscopic MM images may be adjustedvia the input device 1410 to optimize clarity of the fused image. Theexample procedure accordingly enables a safer, more accurate andefficient excision of a tumor.

In some embodiments, the procedure 5200 may be repeated if a FOV, focalpoint, working distance, and/or magnification changes. In theseembodiments, the processor 4102 is configured to use the updated poseinformation and extract the corresponding stereoscopic axis and IPD froma lookup table to re-render the MM data into an updated, accuratestereoscopic image. The processor 4102 is configured to fuse the newlyrendered MM data into the current stereoscopic images such that the liveview and the corresponding MM data are located in the proper position,depth, and orientation.

In some embodiments, the example processor 4102 is configured to operatewith the stereoscopic visualization camera 300, the robotic arm 506,and/or the coupling plate 3304 to generate live cross-sectional fusedvisualizations. A cross-section visualization of a surgical siteprovides a surgeon a significantly improved viewpoint that is nototherwise available. FIG. 53 shows a diagram of a patient 5300 with aglioblastoma 5302, which is illustrated in phantom inside of a patient'shead. Specifically, the glioblastoma 5302 may be located in thepatient's brain 5304, which is shown in light phantom lines. The diagramof FIG. 53 is typical of pre-operative diagnostic images, for example,from an Mill device, where numerous image slices are stacked and a 3Dmodel of an interior of the patient's cranium 5306 is rendered andvisualized.

In the illustrated example, the glioblastoma 5302 is to be removedthrough brain surgery. FIG. 54 shows a diagram of a perspective view ofthe patient 5300 undergoing a craniotomy procedure 5400 to provideaccess to the cranium 5306. The procedure 5400 also includes braindissection and retraction using surgical instrument 5402. Generally, asurgical access site 5404 is made in a deep conical shape to access theglioblastoma 5302.

FIG. 55 shows a diagram of the stereoscopic visualization platform 516including the stereoscopic visualization camera 300 and the robotic arm506 to visualize the craniotomy procedure 5400, according to an exampleembodiment of the present disclosure. As illustrated, the craniotomyprocedure 5400 is set up such that the robotic arm 506 is positioned toaim the stereoscopic visualization camera 300 through the top of thecranium 5306 along visualization axis 5500 of the conical surgical site5404. A view of the operating surgeon is generally through the top ofthe cranium 506, as shown in FIG. 57. As one can appreciate from FIG. 7,the depth of the surgery and, for example, the tip of the surgicalinstrument 5402 is difficult to see.

The example stereoscopic visualization camera 300, shown in FIG. 55,provides a highly accurate stereoscopic image viewed down the axis 5500of the conical surgical access site. As discussed above, parallaxinformation between left and right views of the camera 300 for allpoints common to both views in the access site are used by the processor4102 to determine a depth of each point from a known reference depth,such as for example, the object plane. In the illustrated example,parallax between the left and right views is equal to a value of ‘0’,which enables the processor 4102 to determine a depth map of each pointin the image. The depth map can be re-rendered by the processor 4102 asif the map was viewed from a different angle. Further, the processor4102 is configured to make at least a portion of the depth maptransparent, upon receiving an instruction from a surgeon and/or anoperator. In the illustrated example, a portion of the depth map belowsection plane AA of FIG. 57 can be made transparent by the processor4102, thereby enabling the processor 4102 to generate a cross-sectionalview of the live surgical access site 5404.

FIG. 56 shows a diagram of a phantom view of the conically shapedsurgical access site 5404. The illustrated surgical access site 5404includes stepped conical segments for clarity in this discussion. Inthis example a swept cone angle of the site 5404 is designated by angle‘a’.

FIG. 58 shows a diagram of the conically shaped surgical access site5404 for the craniotomy procedure 5400. Prior knowledge of the size andshape of the surgical instrument 5402, along with image recognition ofits position, direction, and/or orientation enable the processor 4102 togenerate image data for the cross-sectional view shown in FIG. 58.Recognition of the instrument 5402 in the stereoscopic view representedby FIG. 57 enables its precise placement in the cross-sectional view ofFIG. 58 and visualization of, for example, the underside of theinstrument which is not visible to the surgeon while operating on thepatient's brain 5304.

In some embodiments, the processor 4102 is configured to fuse an imageof the glioblastoma 5302 with near-live or live stereoscopic image(s).As discussed above, the combination of the robotic arm 506 and thecamera 300 provides highly accurate position, direction, and/ororientation information of a view with respect to the robot frame ofreference or robot space. After registration or calibration of therobotic arm 506 and the camera 300 to the frame of reference of thepatient 5300, accurate position, direction, and/or orientationinformation of the surgical access site 5404 and its respective positionto the patient is generated by the processor 4102. The processor 4102uses image fusion to superimpose a selection portion of the MRI image ofthe glioblastoma 5302 on to a cross-sectional view, as shown in FIG. 59.In addition, the image fusion enables the visualization of otherrelevant MM image data including, for example, brain vasculature orother structure desired to be included in the image. The exemplarysurgical procedure proceeds with the surgeon being able to see andunderstand the depth location of the glioblastoma 5302 in addition to asafe spacing or positioning of the instrument 5402.

FIG. 59 shows a diagram of a sectional view of the surgical access site5404. In this example, a portion 5302′ of the glioblastoma 5302 isvisible to the stereoscopic visualization camera 300. FIG. 60 shows adiagram of a cross-section view orthogonal to plane AA of FIG. 57. Thediagram may be illustrative of a cross-sectional view generated by theprocessor 4102 based on the MM data fused with the live view of thesurgical access site 5404. The use of the depth map by the processor4102 enables rendering of the surgical access site 5404 at variousdesired section planes and combinations of section planes, as shown inFIG. 61. The rendering enables processor 4102 to display the completeglioblastoma 5302 including the visible portion 5402′ and the reminderfrom the MM data. The processor 4102 may display the visualization froma perspective of the camera 300 or as a cross-sectional view, as shownin FIG. 61.

5. Robotic Motion with Conjoined Visualization Embodiment

In some embodiments, the example processor 4102 operates in connectionwith the robotic arm controller 4106, the stereoscopic visualizationcamera 300, the robotic arm 506, and/or the coupling plate 3304 toconjoin visualization with robotic motion. In some examples, theprocessor 4102 and/or the robotic arm controller 4106 operate in aclosed loop to provide conjoined visualization based on robotic motion.In these examples, the processor 4102 and/or the robotic arm controller4106 are configured to position the robotic arm 506, the coupling plate3304, and/or the camera 300 for visualization of a surgical site basedon a specific image and its contents (e.g., objects, identifiablefeatures, etc.). As discussed above, the robotic arm 506 and camera 300positions are known by the processor 4102 and/or the robotic armcontroller 4106. In addition, image data recorded by the camera isstereoscopic, which provides depth data. As a result, the processor 4102and/or the robotic arm controller 4106 can determine a location on apatient or in robot three-dimensional space of every visualized point.Thus, when the robotic arm 506 moves the camera 300 in a desireddirection from an initial position with an initial image, the desiredimage change is expected to be seen in a second, post-move image.

Alternatively, the expected post-move image can be calculated by theprocessor 4102 and/or the robotic arm controller 4106 being configuredto apply equations representative of the desired move to the initialimage data, which results in a calculated second image. The processor4102 and/or the robotic arm controller 4106 compare the post-move actualimage with the calculated image using a match-template routine orfunction, as described above. If errors are detected, the processor 4102and/or the robotic arm controller 4106 can correct the errors by movingthe robotic arm 506 and/or the camera 300 accordingly. For example,given an initial image and a desired move “100 pixels to the right”received from an operator, the image data for the theoretical movedimage can be calculated as a shift of 100 pixels right by the processor4102 and/or the robotic arm controller 4106. Then, the physical move ismade by performing commands to the various coordinated robot joints, asdisclosed, to relocate the robotic arm 506 and/or the camera 300 to thetheoretical desired location. A second image is recorded by the camera300, which is compared by the processor 4102 and/or the robotic armcontroller 4106 to the calculated image data using, for example a matchtemplate function or its equivalent. If the move is accurate, the datawould indicate a 100% correlation at a tip of the camera 300, where bothimages are perfectly aligned. If, however, the actual image data showsbest correlation at another location, for example 101 pixels right and 5pixels up, then the move could be modified by the processor 4102 and/orthe robotic arm controller 4106 to correct the error by physicallymoving the camera 300, via the robotic arm 506, 1 pixel left and 5pixels down.

6. Sag Compensation Embodiment

In some embodiments, at least some of joints R1 to R9 of the robotic arm506 and/or the coupling plate 3304 may experience some sag. Theprocessor 4102 and/or the robotic arm controller 4106 may be configuredto provide correction for robotic arm sag. In some instances, theprocessor 4102 and/or the robotic arm controller 4106 are configured toperform sag compensation on a series of small moves, such that motionaccuracy is preserved over a range of motion of the robotic arm 506. Forexample, to characterize and eliminate sag, sag compensation isperformed in motion directions that exercise a particular robotic jointto isolate error as a function of actual robot joint rotationalposition. By comparing the error to torque moments calculated bymultiplying camera 300 load weight by moment arm (or link) length, thecompliance of that joint can be determined. Alternatively, jointcompliance may be calculated using analytical techniques, for exampleFinite Element Analysis (“FEA”).

Using and storing the above-compliance characterization for all thejoints in all rotational positions, the processor 4102 and/or therobotic arm controller 4106 may calculate the overall sag for aparticular camera position. The processor 4102 and/or the robotic armcontroller 4106 may determine a sag correction factor for each cameraposition to a LUT and/or calibration registers. Further, the processor4102 and/or the robotic arm controller 4106 may apply the sag correctionfactor to robotic arm move commands or a movement sequence (before orafter scale factors are applied) such that sag compensation isincorporated into movement commands/signals. The correction factor maybe calculated in an ongoing motion procedure, thereby enabling accuratetracking and following of the camera 300. This correction factor furthereliminates a need for a second camera for calibration/positioning of thestereoscopic visualization platform 516, and eliminates the need to havefiducial targets on the camera 300, and hence eliminates a problem ofdrape interference.

7. Storage of Visualization Positions/Orientations Embodiment

In some embodiments, the example processor 4102 is configured to savevisualization parameters to return to a certain orientation and/orposition of the stereoscopic visualization camera 300. The visualizationparameters may include a view vector, location, magnification, workingdistance, focus, position, and/or orientation of the stereoscopicvisualization camera 300, the robotic arm 506 and/or the coupling plate3304.

In an example, a surgeon may wish to have a highly-magnifiedvisualization of a small suture during an anastomosis of a portion of ablood vessel under visual illumination. The surgeon may then zoom-out toa wider view of the entire vessel under infrared illumination to checkfor patency. The surgeon may then return to the magnified visualizationto complete the suture. In this example, the processor 4102 isconfigured to save the visualization parameters at each of thepositions. The processor 4102 may store positions corresponding tolocations that have been continuously viewed for a time period, such astwo seconds, five seconds, thirty seconds, etc. The processor 4102 mayalso store a position after receiving an instruction from the surgeonvia the input device 1410.

The processor 4102 may display a list of stored locations and/orwaypoints. Selection of a stored location causes the processor 4102and/or the robotic arm controller 4106 to move the robotic arm and/orthe coupling plate 3304 to the previous location and adjust opticalparameters, including light illumination and filtering, as setpreviously. Such a configuration enables a surgeon to seamlessly viewall stored locations in sequence without removing their eyes from adisplayed image of the procedure or removing their hands and theirinstruments from the site.

In some embodiments, the processor 4102 may be configured to enable anoperator to create waypoints or positions/orientations prior to asurgical procedure. The waypoints may be provided in a sequence, whichenables the processor 4102 to progress through the specified waypointsduring the procedure after receiving an input from an operator toprogress. The processor 4102 may provide a three-dimensionalrepresentation of the robotic arm 506, the coupling plate 3304, and/orthe camera 300 via the touchscreen input device 1410 a to enable anoperator to virtually position the stereoscopic visualization platform516. This may include providing for a magnification, working distance,and/or focus in relation to a virtualized patient and/or based onalternate modality visualizations of the patient. The processor 4102 isconfigured to store the visualization parameters to, for example, thememory 1570 and/or the memory 4120 for each waypoint.

In some embodiments, the processor 4102 is configured to perform certainvisualizations that are particular to certain procedures. For example,image recognition functionality in the processor 4102 is used toautomatically align the camera 300 with an object of interest. The imageof the surgical site is compared by the processor 4102 to a previousimage or image of the target object to provide for recognition of adesired object and its position and orientation within a stereoscopicimage. The processor 4102 and/or the robotic arm controller 4106 areconfigured to, for example, move the robotic arm 506 toward the objectand zoom the camera 300 towards the object and set the desired imageview attributes for the particular object and procedure. For instance,in ophthalmology, a live retinal image can be compared to a saved imagesuch that, for example, the optic nerve head of the patient's retina canbe located from the image recognition. The processor 4102 and/or therobotic arm controller 4106 then automatically move the robotic arm 506and/or the coupling plate 3304 and focus and/or change a magnificationof the camera 300 such that the tip of the camera 300 is pointed at thenerve head for diagnosis. The processor 4102 may then set the camera 300and/or the monitor 512 for image display without red coloration toenable features of the retina to be more easily distinguished fromsurrounding tissue.

In addition to saving and returning to stored visualizations, paths ofmotion from one view to another can also be saved by the exampleprocessor. In the anastomosis example discussed above, the processor4102 and/or the robotic arm controller 4106 may cause the robotic arm506 and/or the camera 300 to follow an entire length of a blood vesselunder high magnification to check for aneurysms or other conditions. Theprocessor 4102 may be configured to recognize and follow the continuousvessel, as desired. The processor 4102 may perform a match templateroutine on a limited set of pixels to actively determine the directionof motion of the robotic arm 506 and/or the camera 300.

The example processor 4102 may also program and store a path of motionwithin a visualization of an object, made from different viewing angles.For example, an ophthalmological gonioscopy of a patient's eye can beperformed by programming the processor 4102 and/or the robotic armcontroller 4106 to pivot about a point inside the eye. In this example,the robotic arm 506 sweeps the camera 300 in a generally conical motionsuch that the patient's eye is viewed from a plethora of viewing angles.Such motion of surgical site visualizations can be used to select thebest angle to preclude spurious reflections from illumination or to seearound obstructions in alternative viewing angles.

In some embodiments, the processor 4102 is configured to reduceocclusions in depth map calculations. Occlusions are inherent in depthmap calculations due to the parallax of the two views of a stereoscopicimage, where a first view sees some portion of a site different from theother view. As a result, each view does not see some part of the otherview. By moving the robotic arm 506 among various places andrecalculating the depth map while using knowledge of thethree-dimensional locations of the image pixels, occlusion is reduced.The depth map may be made more accurate by iteratively calculating themap after known motion steps are performed, anticipated map changes arecalculated, errors are determined by the difference, and an average mapis constructed.

E. Assisted Drive Embodiments

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 are configured to execute one or more algorithms,routines, etc. defined by instructions stored in the memory 1570 and/or4120 to enable the robotic arm 506 and/or the coupling plate 3304 toprovide powered joint movement based on detected forces applied by anoperator for moving the stereoscopic visualization camera 300. In theseembodiments, the assisted drive feature enables the robotic arm 506 tooperate as an extension of a surgeon by moving the stereoscopicvisualization camera 300 to a desired location and/or orientation. Asdescribed below, the processor 4102 and/or the robotic arm controller4106 are configured to monitor force/torque/movement imparted by anoperator and positions of arm joints to infer an operator's intent andaccordingly move the robotic arm 506 and/or the coupling plate 3304.

FIG. 62 shows a diagram that is illustrative of an algorithm, routine,or procedure 6200 for providing assisted drive of the stereoscopicvisualization camera 300, according to an example embodiment of thepresent disclosure. Although the procedure 6200 is described withreference to the flow diagram illustrated in FIG. 62, it should beappreciated that many other methods of performing the steps associatedwith the procedure 6200 may be used. For example, the order of many ofthe blocks may be changed, certain blocks may be combined with otherblocks, and many of the blocks described are optional. Further, theactions described in procedure 6200 may be performed among multipledevices including, for example the information processor module 1408 ofthe example stereoscopic visualization camera 300 of FIG. 14 and/orjoints R1 to R9 and robotic arm controller 4106 of FIG. 41. In someexamples, the procedure 6200 may be performed by a program stored in thememory 4120 of the robotic arm controller 4106. The example procedure6200 may be executed periodically as force is applied to the camera 300.For example, the procedure 6200 may sample force/torque data everyupdate cycle, which may be 1 (“ms”), 5 ms, 8 ms, 20 ms, 50 ms, etc.

In the illustrated embodiment, the processor 4102 and/or the robotic armcontroller 4106 receive force/torque output data 6201 from the sensor3306 related to force imparted by an operator on the camera 300. Theprocessor 4102 and/or the robotic arm controller 4106 are configured tofilter the received output data 6201 (block 6202). The output data mayinclude a force and/or torque vector. The filtering may include applyinga first low-pass filter, a second low pass filter, and/or a notch filterthat targets cart vibrations. In other examples, a single low-passfilter and a notch filter may be used by the processor 4102 and/or therobotic arm controller 4106.

The example processor 4102 and/or the robotic arm controller 4106 alsoreceive joint position data 6203 from one or more joint sensors in therobotic arm 506 and/or the coupling plate 3304. The processor 4102and/or the robotic arm controller 4106 use the joint position data 6203to provide compensation for the filtered force/torque output data (block6204). The compensation may include gravity compensation and/orforce-application point compensation. For gravity compensation, theeffects of Earth's gravity are removed from the filtered data. Forforce-application point compensation, the processor 4102 and/or therobotic arm controller 4106 provide compensation to the filtered data(and/or gravity compensated data) based on a point where the force wasapplied to the camera 300 (e.g., the control arms 304). As discussedabove in connection with FIG. 35, the sensor 3306 is located some offsetdistance away at an angle from the control arms 304. The offset distanceand angle cause the force applied at the control arms 304 to be slightlyshifted by direction and angle when detected in the sensor 3306. Theforce-application compensation adjusts the force values as though theforce was applied directly to the sensor 3306 instead of the controlarms 304. The force-application compensation may be pre-determined basedon a known angle and/or distance between the sensor 3306 and the controlarms 304. Together, the gravity compensation and the force-applicationpoint compensation modify the filtered force/torque data to create aforce/torque vector that is proportional to the force/torque provided byan operator at the control arms 304 of the camera.

The example processor 4102 and/or the robotic arm controller 4106 alsouse the joint position data 6203 in conjunction with the compensated,filtered force/torque output data to perform a coordinate transformbetween force/torque frame to a global frame or robot space (block6206). The transform may include one or more predefined equations orrelations based on the known robot space and the orientation of thesensor 3306. The example processor 4102 and/or the robotic armcontroller 4106 also use the joint position data 6203 to perform acoordinate transform between a camera frame of the stereoscopicvisualization camera 300 and the global frame or robot space (block6208). The coordinate transform for the camera frame may be based on theoptical calibration parameters mapped to robot space of the robotic arm506, as described above.

After performing the coordinate transforms, the example processor 4102and/or the robotic arm controller 4106 are configured to convert theforce/torque vector(s) into one or more translational/rotational vectorsusing at least one sigmoid function (block 6210). The creation of thetranslational/rotational vector(s) produces an inference of an intendeddirection of the operator. The translational and rotational informationis used to determine how joints of the robotic arm 506 are to be rotatedto mirror, match, and/or approximate the operator's intended movement.

In some examples, the example processor 4102 and/or the robotic armcontroller 4106 are configured to apply robot speed scaling to thetranslational/rotational vector(s) (block 6212). The speed scaling maybe based, for example, on operating conditions of robotic arm 506. Forexample, speed scaling may be applied based, for example, once asurgical procedure has started to prevent the arm from accidentlystriking operating room staff, instruments, and/or the patient at arelatively high rate of speed. When a procedure has not yet begun, theexample processor 4102 and/or the robotic arm controller 4106 may applyless speed scaling for calibration or setting of the robotic arm 506when a patient is not present.

The example processor 4102 and/or the robotic arm controller 4106determine potential movement sequences of joints of the robotic arm 506based on the scaled translational/rotational vector(s). While evaluatingpossible sequences, the processor 4102 and/or the robotic arm controller4106 identify joint singularities for avoidance, thereby ruling out thecorresponding movement operations of the robotic arm 506 (block 6214).As discussed above, singularities may include elbow lock or otherpositions that may be prone to hysteresis and backlash. The processor4102 and/or the robotic arm controller 4106 are configured to select amovement sequence, after movement singularities are eliminated using,for example, Jacobian kinematics (e.g., an inversion of a Jacobianmatrix) (block 6216). The Jacobian kinematic equations define howcertain joints of the robotic arm 506 and/or the coupling plate 506 areto be moved based on the scaled translational/rotational vector(s). TheJacobian kinematics provide for velocity control while inversekinematics, discussed below, provide for positional control. In someembodiments, the processor 4102 and/or the robotic arm controller 4106may instead use inverse kinematics or other robotic arm controlroutines. The processor 4102 and/or the robotic arm controller 4106determine a movement sequence that specifies how certain joints of therobotic arm and/or coupling plate 3304 are to move in a coordinatedmanner and specifies, for example, joint rotation speed, jointrotational direction, and/or joint rotational duration. The movementsequence may also specify a sequence in which joints of the robotic arm506 and/or the coupling plate 3304 are to be rotated. Any of joints R1to R9 of the robotic arm and/or coupling plate 3304 may rotateindividually or have overlapping movement depending on the movementsequence.

After a movement sequence is determined, the processor 4102 and/or therobotic arm controller 4106 are configured to perform collisionavoidance using joint speed scaling and/or boundaries. For example, theprocessor 4102 and/or the robotic arm controller 4106 are configured todetermine if the movement sequence will cause one or more joints and/orlinks of the robotic arm 506 and/or the coupling plate 3304 to approacha boundary or other defined Cartesian limit, such as space around apatient or instrument. As discussed above in connection with FIG. 49,the processor 4102 and/or the robotic arm controller 4106 may compareestimates of positions of the links and/or joints in the robot spacefrom the movement sequence to one or more defined boundaries and/orangle limits. Based on a distance from a boundary, the processor 4102and/or the robotic arm controller 4106 applies one or more joint speedlimits via a scale value (block 6218). The processor 4102 and/or therobotic arm controller 4106 may also apply one or more joint positionlimits (block 6220) that prevent, for example, links of the robotic arm506 from striking each other and/or prevent the robotic arm 506, thecoupling plate 3304, and/or the camera 300 from extending past aboundary. Locations just before position limits (e.g., 1 centimeter(“cm”) 2 cm, 10 cm, etc. before a position limit) and/or locations atthe position limits may correspond to locations in Cartesian robot spacewhere a value of the scale factor is ‘0’.

In some examples, the processor 4102 and/or the robotic arm controller4106 may perform Jacobean kinematics with the boundaries provided as aninput to the equations, where movement through areas close to a boundaryare provided a higher cost factor. The use of boundary cost factorscauses the processor 4102 and/or the robotic arm controller 4106 toavoid locations close to boundaries, if possible, when determining amovement sequence. The cost factor may include inversely proportional toa decrease in a scale factor associated with a particular location inrobot space. The scale factor may apply to each joint/link, or separatescale factors may exist for each joint for the same location in robotspace.

After providing for collision avoidance, the example processor 4102and/or the robotic arm controller 4106 are configured to provide forcorrection for relatively fast reversals of the robotic arm 506 (block6222). The processor 4102 and/or the robotic arm controller 4106 mayimplement a zero phase delay algorithm to reject directional impulsesthat quickly cause one or more joints to change rotational direction.The zero phase delay algorithm may be implemented by a filter thatprevents, for example, the robotic arm from bucking or rocking if anoperator reverses direction too quickly.

As illustrated in FIG. 62, the example processor 4102 and/or the roboticarm controller 4106 are configured to validate commands of the movementsequence (block 6224). The processor 4102 and/or the robotic armcontroller 4106 may validate a command to ensure that a command (orsignal indicative of a command) is within operating parameters (e.g.,duration, rotational speed, etc.) of a joint motor. The processor 4102and/or the robotic arm controller 4106 may also validate a command bycomparing the command to current thresholds to ensure the robotic arm506 will not draw excess current during any phase of the movementsequence.

The example processor 4102 and/or the robotic arm controller 4106 mayalso apply one or more anti-noise filters to the movement commands orsignals indicative of the movement commands (block 6226). The filter mayinclude a high frequency low-pass filter that removes high frequencynoise components, which may induce transient signals in a joint motor.After any filtering, the processor 4102 and/or the robotic armcontroller 4106 transmit the one or more commands via one or moresignals or messages to the appropriate joint motor of the robotic arm506 and/or the coupling plate 3304 according to the movement sequence(block 6228). The transmitted commands cause motors at the respectivejoints to move the robotic arm 506 and/or the coupling plate 3304,thereby causing the camera 300 to move as intended by the operator. Theexample procedure 6200 may repeat as long as an operator applies forceto the camera 300.

FIG. 63 shows a diagram of an example procedure 6300 for moving theexample visualization camera 300 using an input device 1410, accordingto an example embodiment of the present disclosure. The exampleprocedure 6300 is nearly identical to the procedure 6200 of FIG. 62,except blocks 6202 to 6206 related to the sensor 3306 are removed. Inthe illustrated example, a control input 6301 is received from an inputdevice 1410, such as buttons on the control arm 304, a foot pedal,joystick, touchscreen interface, etc. The control input 6301 isindicative of directional movement of the camera in the Cartesian robotspace of the robotic arm 506.

As illustrated in FIG. 63, the control input 6301 is combined with thejoint position data 6203 from one or more joint sensors in the roboticarm 506 and/or the coupling plate 3304 for performing a coordinatetransform from a camera frame to a global frame and/or robot space(block 6208). The example procedure 6300 then continues in the samemanner as discussed for procedure 6200. The processor 4102 and/or therobotic arm controller 4106 accordingly cause the robotic arm 506, thecoupling plate 3304, and/or the camera 300 to move to a desired locationand/or orientation based on the control input 6301 received from theinput device 1410.

F. Lock-to-Target Embodiments

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 are configured to execute one or more algorithms,routines, etc. defined by instructions stored in the memory 1570 and/or4120 to enable the robotic arm 506 and/or the coupling plate 3304 toprovide a lock-to-target feature. In these embodiments, thelock-to-target feature enables the robotic arm 506 to operate as anextension of a surgeon by enabling the stereoscopic visualization camera300 to be re-oriented while being locked onto a target surgical site. Asdescribed below, the processor 4102 and/or the robotic arm controller4106 are configured to monitor force/torque/movement imparted by anoperator and positions of arm joints to infer an operator's intent andaccordingly re-orientate the robotic arm 506 and/or the coupling plate3304 such that the focal point of the camera 300 remains locked orstationary.

The lock-to-target feature enables the camera 300 to be reoriented bycausing all motion to be constrained to the surface of a virtual sphere.The tip of the camera 300 is located at an outer surface of the virtualsphere (e.g., a top hemisphere of the virtual sphere) and a focal pointof the camera 300 or target surgical site constitutes a center of thevirtual sphere. The example processor 4102 and/or the robotic armcontroller 4106 enable an operator to move the camera 300 over an outersurface of the virtual sphere while keeping the camera 300 pointed atthe center of the sphere, thereby keeping the target surgical site infocus during the movement. The lock-to-target feature enables anoperator to easily and quickly obtain significantly different views ofthe same target site.

FIG. 64 shows a diagram that is illustrative of an algorithm, routine,or procedure 6400 for providing a lock-to-target for the stereoscopicvisualization camera 300, according to an example embodiment of thepresent disclosure. Although the procedure 6400 is described withreference to the flow diagram illustrated in FIG. 64, it should beappreciated that many other methods of performing the steps associatedwith the procedure 6400 may be used. For example, the order of many ofthe blocks may be changed, certain blocks may be combined with otherblocks, and many of the blocks described are optional. Further, theactions described in procedure 6400 may be performed among multipledevices including, for example the information processor module 1408 ofthe example stereoscopic visualization camera 300 of FIG. 14 and/orjoints R1 to R9 and robotic arm controller 4106 of FIG. 41. In someexamples, the procedure 6400 may be performed by a program stored in thememory 4120 of the robotic arm controller 4106.

The example procedure 6400 is similar to the assisted drive procedure6200. However, the procedure 6400 provides for the commanding of jointpositions to retain a focal point of the camera 300 while the exampleprocedure 6200 provides for the calculation of joint velocities. Theexample procedure 6400 determines a desired force/movement vector inputby an operator and calculates a rotational transform such that the focalpoint of the camera 300 remains stationary while one or more joints ofthe robotic arm 506 and/or the coupling plate 3304 are moved tore-orient the camera 300. The reorientation of the camera 300 enables atarget surgical site to be imaged from different angles. Thereorientation may be needed when a first view path is blocked by, forexample, an instrument, and the surgeon desires to maintain the currentfocal point.

The example procedure 6400 begins when an operator selectslock-to-target button on the input device 1410, which causes aninstruction message or signal to be transmitted to the processor 4102and/or the robotic arm controller 4106. After receiving the message, theprocessor 4102 and/or the robotic arm controller 4106 operate in alock-to-target mode where the working distance and/or focal point isheld stationary while enabling an operator to change an orientation ofthe camera 300, which causes one or more joints of the robotic armand/or coupling plate 3304 to provide assisted movement. When aninstruction is received, the example processor 4102 and/or the roboticarm controller 4106 may record the current working distance,magnification, focus, and/or other optical parameters of the camera 300.The processor 4102 and/or the robotic arm controller 4106 may alsorecord a current image of the FOV.

After the procedure 6400 begins, the processor 4102 and/or the roboticarm controller 4106 receive force/torque output data 6201 from thesensor 3306 related to force imparted by an operator on the camera 300.As discussed in connection with FIG. 62, the processor 4102 and/or therobotic arm controller 4106 filter and provide gravity/force-applicationcompensation for the data 6102 (blocks 6202 and 6204). Also similar toFIG. 62, the processor 4102 and/or the robotic arm controller 4106 usethe joint position data 6203 in conjunction with the compensated,filtered force/torque output data to perform a coordinate transformbetween force/torque frame to a global frame or robot space (block6206). The example processor 4102 and/or the robotic arm controller 4106also use the joint position data 6203 to perform a coordinate transformbetween a camera frame of the stereoscopic visualization camera 300 andthe global frame or robot space (block 6208). The example processor 4102and/or the robotic arm controller 4106 also perform a transform from theglobal frame or robot space to spherical coordinates that correspond toa virtual sphere (block 6410).

After the coordinate transforms, the example processor 4102 and/or therobotic arm controller 4106 are configured to scale trajectory speedbased, for example, on an operation mode of the camera 300 (block 6412).The scaling may be similar to the scaling performed at block 6212 ofFIG. 62. The example procedure 6400 of FIG. 64 continues by theprocessor 4102 and/or the robotic arm controller 4106 calculating asphere end point (block 6414). Calculation of the sphere end pointprovides an inference about the operator's desired movement directionand determines how far the camera 300 is to be moved over the virtualsphere without rotating the sphere.

FIG. 65 shows a diagram that is illustrative of a virtual sphere 6500for the lock-to-target feature, according to an example embodiment ofthe present disclosure. As shown in FIG. 65, the stereoscopicvisualization camera 300 is virtually placed on the sphere 6500 based ona current position, as determined from the joint position data 6203. Aview vector of the camera 300 points to a tip, designated as the xyztarget, which is located in a center of the sphere 6500. The processor4102 and/or the robotic arm controller 4106 are configured to use thetransformed force/torque data to determine how the camera 300 on thesphere is to move along a surface of the sphere 6500 while maintainingthe view vector pointed at the xyz target, where any given point on thesphere is given by an equation that is a function of rotational sphereangles v′ and ‘u’. When the force/torque data is used, the processor4102 and/or the robotic arm controller 4106 use an ‘x’ and ‘y’ componentcorresponding to the translational force for directly determining howthe camera 300 is to move on the virtual sphere 6500 to determine thesphere end point.

The processor 4102 and/or the robotic arm controller 4106 may determinethe sphere end point differently for different inputs. For example, ifan input is received via the input device 1410, as shown in FIG. 63, theprocessor 4102 and/or the robotic arm controller 4106 converts ‘up’,‘down’, ‘left’, and ‘right’ from camera coordinates to robot spacecoordinates, which are provided as x,y vectors. Similar to theforce/torque data, the x,y vectors are used by the processor 4102 and/orthe robotic arm controller 4106 for directly determining how the camera300 is to move on the virtual sphere 6500 to determine the sphere endpoint. It should be appreciated that in instances where inputs arereceived via the input device, the operations discussed in conjunctionwith blocks 6202 to 6206 may be omitted, as shown in FIG. 63.

In some examples, the processor 4102 and/or the robotic arm controller4106 are configured to receive orbit input data. In these examples, theprocessor 4102 and/or the robotic arm controller 4106 hold the sphereangle constant while iterating movement along sphere angle ‘u’ of thevirtual sphere 6500. The iterative movement along sphere angle ‘u’enables the sphere end point to be determined for the orbit input. Itshould be appreciated that while the inputs are applied to the virtualsphere 6500, in other examples, the inputs may be applied to othershapes. For example, the virtual sphere 6500 instead may be defined as avirtual cylinder, an ellipsoid, an egg-shape, a pyramid/frustum, etc.

In other examples, the processor 4102 and/or the robotic arm controller4106 are configured to receive level scope input data. In theseexamples, the processor 4102 and/or the robotic arm controller 4106 holdthe sphere angle ‘u’ constant while iterating movement along sphereangle ‘v’ of the virtual sphere 6500. The iterative movement alongsphere angle ‘v’ causes the camera 300 to be moved to a top of thevirtual sphere 6500.

Returning to FIG. 64, after the sphere end point is determined, theprocessor 4102 and/or the robotic arm controller 4106 are configured tocalculate an amount of rotation needed for the camera 300 to maintainthe lock at the x,y,z target after the camera 300 has been moved alongthe virtual sphere to the determined end point (block 6416). Theprocessor 4102 and/or the robotic arm controller 4106 may also provideanti-yaw correction during this calculation (block 6418). In otherwords, the processor 4102 and/or the robotic arm controller 4106 areconfigured to determine how the camera 300 is to be orientated given itsnew position on the virtual sphere 6500 such that the view vector or tipof the camera 300 is provided at the same x,y,z target, which is pointedat a center of the virtual sphere 6500, which corresponds to a targetsurgical site or focal point.

During this step, the processor 4102 and/or the robotic arm controller4106 determine the joint angles of the robotic arm 506 and/or thecoupling plate 3304 needed to achieve the desired orientation. After thex,y,z sphere end point is calculated in block 6414, the processor 4102and/or the robotic arm controller 4106 determine roll and pitch amountsfor the camera 300. In some embodiments, the calculation is a two-stepprocess. First, the processor 4102 and/or the robotic arm controller4106 calculate an initial 4×4 transform matrix T that provides movementof the camera 300 without rotation given the x,y,z sphere end point.Then, the processor 4102 and/or the robotic arm controller 4106calculate local roll and pitch amounts such that the camera 300 remainslocked at a target located at x,y,z (and/or positioned at the x,y,zsphere end point) for subsequent cycles of joint rotations. Theprocessor 4102 and/or the robotic arm controller 4106 may use Equations(4) and (5) below to calculate roll and pitch amounts, where T nextcorresponds to a 4×4 transform matrix. The calculations can be performedat each update cycle (e.g., 8 ms).

$\begin{matrix}{T_{next} = {T\begin{pmatrix}R & 0 \\0 & 1\end{pmatrix}}} & (4)\end{matrix}$

-   -   such that:    -   x_(target_next)=x_(target)    -   y_(target_next)=y_(target)    -   z_(target_next)=z_(target)

$\begin{matrix}{R = {\begin{bmatrix}{\cos\mspace{11mu}} & 0 & {\sin\mspace{11mu}} \\0 & 1 & 0 \\{{- \sin}\mspace{11mu}} & 0 & {\cos\mspace{11mu}}\end{bmatrix}\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\mspace{11mu}} & {{- \sin}\mspace{11mu}} \\0 & {\sin\mspace{11mu}} & {\cos\mspace{11mu}}\end{bmatrix}}} & (5)\end{matrix}$

In Equation (4) above, x_(taregt_next), y_(taregt_next), andz_(taregt_next) are constraints on the T_(next) matrix. Theabove-constraints specify that the roll and pitch angles are chosen suchthat the x,y,z equations above are valid. In other words, the x,y,zlocation of a target at a next update cycle of joint rotations has to beequal to the x,y,z location of the target in the current cycle. Theconstraints enable the camera 300 to be rotated via roll and pitchangles but remained locked relative to the x,y,z location.

Further, −sin θ on the bottom row of the first matrix of Equation (5)corresponds to a pitch angle while sin θ on the bottom row of the secondmatrix corresponds to a roll angle. A closed form expression for pitchmay exist given function cos(roll). The processor 4102 and/or therobotic arm controller 4106 may use an iterative method to estimateroll, calculated as function cos(roll), with pitch equal tofn(cos(roll)) to generate a correct roll/pitch solution pair for theequations above.

After the roll and pitch amounts are calculated from the operationsdescribed in connection with blocks 6416 and 6418, the example processor4102 and/or the robotic arm controller 4106 are configured to providesingularity avoidance and calculate inverse kinematics to determinejoint rotation to achieve the roll and pitch amounts in addition to thenew x,y,z position of the camera 300 along the virtual sphere 6500(blocks 6214 and 6420). The calculation of the inverse kinematicsenables the processor 4102 and/or the robotic arm controller 4106 todetermine a movement sequence for joints of the robotic arm 506 and/orthe coupling plate 3304.

The example processor 4102 and/or the robotic arm controller 4106 mayapply error correction for the movement sequence in addition to jointspeed limits and/or position limits (blocks 6418, 6218, 6220). Asdiscussed above in connection with FIG. 62, the limits and errorcorrection may prevent the robotic arm 506, the camera 300, and/orcoupling plate 3304 from hitting themselves, exceeding one or moreboundaries, and/or being within acceptable joint positions. Theprocessor 4102 and/or the robotic arm controller 4106 may also validatecommands for the joints of the movement sequence provide anti-noisefiltering before sending the commands (or signals indicative of thecommands) to one or more joints R1 to R9 of the robotic arm 506 and/orthe coupling plate 3304 based on the movement sequence (blocks 6224,6226, 6228). The example procedure 6400 may then end if no othermovement is detected. Otherwise, the procedure 6400 is repeated atperiodic intervals (e.g., 10 ms, 20 ms, etc.) as operator inputs arereceived.

In some embodiments, the processor 4102 and/or the robotic armcontroller 4106 may provide lock-to-target tracking for instruments. Inthese examples, the xyz target of a center of the virtual sphere 6500 isreplaced with a dynamic trajectory that corresponds to a moving target.Such a feature may enable a tracking of spinal tools, for example. Inthese embodiments, an instrument may include one or more fiducialsand/or other markers. The example stereoscopic visualization camera 300records images that include the fiducials. The processor 4102 and/or therobotic arm controller 4106 may perform a coordinate transform from thecamera frame space to robot space to determine how the instrument isbeing moved along the x,y,z axes. The example processor 4102 and/or therobotic arm controller 4106 track how the fiducials move in the imageand determine the corresponding x,y,z movement vectors. In someinstances, the x,y,z vectors may be input into the sphere end pointcalculation of block 6414 of FIG. 64 to change the location of a centerof the virtual sphere 6500. In response to a movement of the sphere6500, the processor 4102 and/or the robotic arm controller 4106determine how the robotic arm 506 is to be positioned to maintain thesame working distance and/or orientation with the new target location.The processor 4102 and/or the robotic arm controller 4106 may then applyinverse kinematics to determine joint rotations of the robotic arm 506and/or the coupling plate to track the movement of the target. Similarto the procedures 6200 and 6400, the processor 4102 and/or the roboticarm controller 4106 may apply error correction, joint limits, filters,and/or validation before sending commends to joints as specified in adetermined movement sequence.

CONCLUSION

It will be appreciated that each of the systems, structures, methods andprocedures described herein may be implemented using one or morecomputer programs or components. These programs and components may beprovided as a series of computer instructions on any conventionalcomputer-readable medium, including random access memory (“RAM”), readonly memory (“ROM”), flash memory, magnetic or optical disks, opticalmemory, or other storage media, and combinations and derivativesthereof. The instructions may be configured to be executed by aprocessor, which when executing the series of computer instructionsperforms or facilitates the performance of all or part of the disclosedmethods and procedures.

It should be understood that various changes and modifications to theexample embodiments described herein will be apparent to those skilledin the art. Such changes and modifications can be made without departingfrom the spirit and scope of the present subject matter and withoutdiminishing its intended advantages. It is therefore intended that suchchanges and modifications be covered by the appended claims. Moreover,consistent with current U.S. law, it should be appreciated that 35U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, paragraph 6 is not intended tobe invoked unless the terms “means” or “step” are explicitly recited inthe claims. Accordingly, the claims are not meant to be limited to thecorresponding structure, material, or actions described in thespecification or equivalents thereof

The invention is claimed as follows:
 1. A robotic imaging apparatuscomprising: a base section configured for connection to a securestructure or a cart; a robotic arm including a first end connected tothe base section, a second end including a coupling interface, and aplurality of joints and links connecting the first end to the secondend, each joint including a motor configured to rotate the joint aroundan axis and a joint sensor configured to transmit a position of therespective joint; a stereoscopic camera connected to the robotic arm atthe coupling interface, the stereoscopic camera configured to recordleft and right images of a target surgical site for producing a streamof stereoscopic images of the target surgical site; a sensor positionedat the coupling interface and configured to detect and transmit outputdata that is indicative of translational and/or rotational forceimparted on the stereoscopic camera by an operator; a memory storing atleast one algorithm defined by one or more instructions and/or datastructures that specify a rotation direction, speed, and duration ofmovement for each of the joints of the robotic arm based at least on acurrent position of the robotic arm and detected translational and/orrotational forces; and at least one processor communicatively coupled tothe sensor and the robotic arm, the at least one processor configuredto: receive the output data from the sensor that is indicative of thetranslational and/or rotational forces, determine, using the at leastone algorithm in the memory, a movement sequence for the robotic armbased on a current position of the robotic arm and the output data fromthe sensor, and cause at least one of the joints of the robotic arm torotate based on the determined movement sequence via one or more motorcontrol signals provided to the at least one joint, wherein the rotationof the at least one joint provides power-assisted movement of therobotic arm based on the detected translational and/or rotational forcesimparted on the stereoscopic camera by the operator.
 2. The apparatus ofclaim 1, wherein the at least one processor is configured to determinethe current position of the robotic arm based on output data from thejoint sensors of the plurality of joints.
 3. The apparatus of claim 1,wherein the sensor includes at least one of a six-degrees-of-freedomhaptic force-sensing device or a torque sensor.
 4. The apparatus ofclaim 1, wherein the stereoscopic camera includes at least one controlarm having a release button to enable the power-assisted movement, andwherein the at least one processor is configured to: receive an inputmessage indicative that the release button was selected; and determinethe movement sequence using the output data from the sensor afterreceiving the input message related to the release button.
 5. Theapparatus of claim 1, further comprising a coupling plate with a firstend configured to connect to the coupling interface of the robotic armand a second end including a second coupling interface configured toconnect to the stereoscopic camera, wherein the coupling plate includesat least one joint including a joint sensor configured to transmit aposition of the respective joint and a motor that is controllable by theat least one processor according to the movement sequence.
 6. Theapparatus of claim 5, wherein the sensor is located at the couplinginterface or the second coupling interface.
 7. The apparatus of claim 5,wherein the coupling plate includes a second joint that enables thestereoscopic camera to be manually rotated by an operator between ahorizontal orientation and a vertical orientation, wherein the secondjoint includes a joint sensor configured to transmit a position of thesecond joint.
 8. The apparatus of claim 1, wherein the stereoscopiccamera includes a housing including a bottom side that is configured toconnect to the robotic arm at the coupling interface.
 9. The apparatusof claim 1, wherein the processor is configured to: determine a leastone scale factor based on at least one of the current position of therobotic arm or a future position of the robotic arm based on themovement sequence; and apply the scale factor to at least one jointspeed of the movement sequence.
 10. The apparatus of claim 9, whereinthe at least one scale factor is configured based on a distance of therobotic arm or the stereoscopic camera from a virtual boundary, andwherein the at least one scale factor decreases to a value of ‘0’ as thevirtual boundary is approached.
 11. The apparatus of claim 10, whereinthe processor is configured to cause a display device to display an iconindicative that the at least one scale factor has been applied to themovement sequence.
 12. The apparatus of claim 1, wherein the processoris configured to: determine a least one scale factor based on jointangles between joints of the robotic arm or joint limits; and apply thescale factor to at least one joint speed of the movement sequence. 13.The apparatus of claim 1, wherein the processor is configured to:provide gravity compensation for the output data; and provideforce-application compensation for the output data to compensate for anoffset between a location of the sensor and a location of thestereoscopic camera upon which the translational and/or rotational forceis imparted by the operator.
 14. The apparatus of claim 1, wherein theprocessor is configured to: determine or identify joint singularitiesfor the plurality of joints of the robotic arm for control of hysteresisand backlash; and determine the movement sequence based on the at leastone algorithm while avoiding robotic arm movement through the jointsingularities.
 15. The apparatus of claim 1, wherein the processor isconfigured to cause at least one of the joints of the robotic arm torotate by transmitting one or more command signals to the motor of therespective joint indicative of the rotation direction, the speed, andthe duration of movement as specified by the movement sequence.
 16. Theapparatus of claim 1, wherein the processor is configured to compareimages recorded by the stereoscopic camera as the robotic arm is beingmoved during the movement sequence to confirm the robotic arm is beingmoved as determined during the movement sequence.
 17. The apparatus ofclaim 1, wherein the at least one algorithm includes use of at least oneof inverse kinematics and Jacobean kinematics.
 18. A robotic imagingapparatus comprising: a robotic arm including a first end configured forconnection to a secure structure, a second end including a couplinginterface, and a plurality of joints and links connecting the first endto the second end, each joint including a motor configured to rotate thejoint around an axis and a joint sensor configured to transmit aposition of the respective joint; a stereoscopic camera connected to therobotic arm at the coupling interface, the stereoscopic cameraconfigured to record left and right images of a target surgical site forproducing a stream of stereoscopic images of the target surgical site; asensor positioned at the coupling interface and configured to detect andtransmit output data that is indicative of translational and/orrotational force imparted on the stereoscopic camera by an operator; amemory storing at least one algorithm defined by one or moreinstructions and/or data structures that specify a rotation direction,speed, and duration of movement for each of the joints of the roboticarm based at least on a current position of the robotic arm and detectedtranslational and/or rotational forces; and at least one processorcommunicatively coupled to the sensor and the robotic arm, the at leastone processor configured to: receive the output data from the sensorthat is indicative of the translational and/or rotational forces,determine, using the at least one algorithm in the memory, a movementsequence for the robotic arm based on a current position of the roboticarm and the output data from the sensor, and cause at least one of thejoints of the robotic arm to rotate based on the determined movementsequence via one or more motor control signals provided to the at leastone joint.
 19. The apparatus of claim 18, wherein the rotation of the atleast one joint provides power-assisted movement of the robotic armbased on the detected translational and/or rotational forces imparted onthe stereoscopic camera by the operator.
 20. A robotic imaging apparatuscomprising: a robotic arm including a first end configured forconnection to a secure structure, a second end including a couplinginterface, and a plurality of joints and links connecting the first endto the second end, each joint including a motor configured to rotate thejoint around an axis and a joint sensor configured to transmit aposition of the respective joint; a stereoscopic camera connected to therobotic arm at the coupling interface, the stereoscopic cameraconfigured to record left and right images of a target surgical site forproducing a stream of stereoscopic images of the target surgical site; asensor positioned at the coupling interface and configured to detect andtransmit output data that is indicative of translational and/orrotational force imparted on the stereoscopic camera by an operator; andat least one processor communicatively coupled to the sensor and therobotic arm, the at least one processor configured to: receive theoutput data from the sensor that is indicative of the translational androtational forces, determine, using at least one algorithm defined byone or more instructions and/or data structures stored in a memory, amovement sequence of rotation direction, speed and duration for each ofthe joints of the robotic arm based on a current position of the roboticarm and the output data from the sensor, and cause at least one of thejoints of the robotic arm to rotate based on the determined movementsequence via one or more motor control signals provided to the at leastone joint.